Heflin first rose to political prominence as a delegate who helped to draft the 1901 Alabama state constitution. Heflin argued, successfully, for completely excluding Black Alabamians from voting, stating that he truly believed that "God Almighty intended the negro to be the servant of the white man." As Secretary of State in 1903, Heflin was an outspoken supporter of men put on trial for enslaving African American laborers through fraudulent convict leasing. As detailed in Douglas A. Blackmon's book, Slavery by Another Name, these practices were a brutal, post-emancipation form of slavery in which African Americans were often illegally convicted of crimes and then sold to farmers or industrialists. Heflin explicitly used white supremacist rhetoric to mobilize support for the defendants. He argued before a group of Confederate veterans that forcing African Americans to labor was a means to hold them in their proper social position.
In 1904, Heflin was elected to the United States House of Representatives as a Democrat to fill the vacancy left by the death of Charles Winston Thompson. Four years later, while a member of the House, he shot and seriously wounded a black man who confronted him on a Washington streetcar. Heflin threw Lewis Lundy, a black man, off the streetcar and shot at him through the streetcar window. Lundy received a wound to the head, reports vary on whether it was due to pistol-whipping by Heflin, by the fall from the streetcar, or by a bullet wound. A white bystander, Thomas McCreary, was wounded by a stray bullet fired by Helflin. Although indicted, Heflin had the charges dismissed. In subsequent campaigns, he bragged of the shooting as one of his major career accomplishments.
He continued to serve in the House until 1920, when he was elected to the Senate to fill the vacancy caused by the death of John H. Bankhead. In the 1920s, he expressed strong hostility to the Knights of Columbus. In 1928, Heflin further expressed outrage that Al Smith was the party's nominee and inveighed against Catholic influences on the Democratic Party. Hence instead of Smith he supported RepublicanHerbert Hoover for President and is sometimes credited with coining the term yellow dog. The Democrats thus did not renominate Heflin for the Senate in 1930.
He ran as an independent candidate, losing decisively to John H. Bankhead II. Returning to Washington to serve out his term, Heflin initiated a Senate investigation of voting fraud to try to overturn Bankhead's election. The inquiry lasted fifteen months and cost $100,000.
In that same year, James Heflin officially protested in the Senate against New York's legalization of racial intermarriage between a black man and a white woman. New York senator Royal S. Copeland reacted angrily to Heflin, who replied that if Copeland went someday to the South on a presidential campaign, he would be lynched and hanged by the population.
In April 1932, with Heflin's term expired and Bankhead seated, the Senate prepared to vote on a committee recommendation against Heflin. Heflin delivered a five-hour oration, punctuating his remarks with vehement gestures and racist jokes. As he thundered to a conclusion, the gallery audience, packed with his supporters, jumped to its feet with a roar of approval. They were ordered out of the chamber. Two days later, the Senate voted by a wide margin to dismiss Heflin's claim.