Smith moved to Nashville, Tennessee in 1951 where he became pastor of First Baptist Church, Capitol Hill, a post he would retain until his death in 1984. He became president of the Nashville NAACP in 1956 and founded the Nashville Christian Leadership Council (NCLC) in 1958. Through the NCLC, Smith helped to organize and support the Nashville sit-ins—a movement which would successfully end racial segregation at lunch counters in Nashville. In a 1964 interview with Robert Penn Warren for the book Who Speaks for the Negro?, Smith comments that the end to segregation was achieved through much hardship and many negotiations by the NCLC.
Smith was married to Alice Clark Smith and had four children, daughters Joy Ardelia, Adena Modesta, and Valerie Lin, and son Kelly Miller Smith Jr. He and his wife also reared a foster daughter Dorothy Jean Springfield.
Kelly Miller Smith interviews housed in the website Who Speaks for the Negro? hosted by Robert Penn Warren Center for the Humanities at Vanderbilt University. The website is a digital archive of materials related to the book of the same name published by Robert Penn Warren in 1965. The original materials are held at the University of Kentucky and Yale University Libraries. The archive consists of digitized versions of the original reel-to-reel recordings that Warren compiled for each of his interviewees as well as print materials related to the project. Digital archive created and designed by the Robert Penn Warren Center for the Humanities at Vanderbilt University.
The Kelly Miller Smith Papers housed at Vanderbilt Library in Special Collections, include his sermons, private correspondence, as well as Smith's musical compositions and other papers. A Finding Aid is available for this collection and the later Kelly Miller Smith Papers Addition.