Williams was elected to the House of Representatives in a special election in 1953, and was re-elected in 1954 but defeated for re-election in 1956. He was elected to the Senate in 1958 and re-elected in 1964, 1970 and 1976. In 1976, he defeated David A. Norcross, who went on to serve the Republican Party in state and national leadership roles.
In 1980, Williams, a resident of Westfield, New Jersey at the time, was convicted of bribery and conspiracy in the Abscam scandal for taking bribes in a sting operation by the Federal Bureau of Investigation (FBI). The Senate Committee on Ethics recommended that Williams be expelled because of his "ethically repugnant" conduct. Prior to a Senate vote on his expulsion, Williams resigned on March 11, 1982. Sentenced to three years, he served two years in Federal prison as Inmate #06089-050, the first time in over 80 years that a senator had spent time in prison. Williams was also charged a $50,000 fine in addition to prison time. Released on January 31, 1986, he served the remainder of his sentence at the Integrity House halfway house, where he later became a member of the board of directors until his death on November 17, 2001. He also attempted to receive a presidential pardon from President Bill Clinton, but his request was denied.
The Metropark train station had been renamed Harrison A. Williams Metropark Station in 1979, in recognition of his support for its construction. However, the name was removed from the station after his conviction.