In 1918, Warburg married composer and musician Kay Swift. Swift was a Protestant and Warburg's uncle, Jacob Schiff, objected to the marriage. They divorced in late 1934 as a consequence of her long involvement with George Gershwin. They had three daughters: April Gagliano, Andrea Kaufman, and Kay Levin. Warburg married his second wife, Phyllis Baldwin, in 1935. He and his third wife, Joan Melber, were married on September 6, 1948, and had four children: James P., Jr., Jennifer, Philip, and Sarah Bliumis Dunn. Warburg is also the grandfather of novelist Katharine Weber.
Under the pseudonym Paul James he wrote the lyrics to Swift's 1929 hit song "Can't We Be Friends?" and their 1930 musical, Fine and Dandy, which introduced the song "Fine and Dandy".
^Senate Report (Senate Foreign Relations Committee) (1950). Revision of the United Nations Charter: Hearings Before a Subcommittee of the Committee on Foreign Relations, Eighty-First Congress. United States Government Printing Office. p. 494.Testimony on Wikisource