Rose was born in Henderson, North Carolina, the only child of Margaret (née Frazier) and Charles Peete Rose, Sr., tobacco farmers who owned a country store. As a child, Rose lived above his parents' store in Henderson and helped out with the family business from age seven. Rose admitted in a Fresh Dialogues interview that as a child his insatiable curiosity was constantly getting him in trouble. A high school basketball star, Rose entered Duke University intending to pursue a degree with a pre-med track, but an internship in the office of Democratic North Carolina Senator B. Everett Jordan got him interested in politics. Rose graduated in 1964 with a bachelor's degree in history. At Duke, he was a member of the Kappa Alpha Order fraternity. He earned a Juris Doctor from the Duke University School of Law in 1968. He met his wife, Mary (King), while attending Duke.
After his wife was hired by the BBC (in New York), Rose handled some assignments for the BBC on a freelance basis. In 1972, while working at New York bank Bankers Trust, Rose landed a job as a weekend reporter for WPIX-TV. His "break" came in 1974, after Bill Moyers hired Rose as managing editor for the PBS series Bill Moyers' International Report. In 1975, Moyers named Rose executive producer of Bill Moyers Journal. Rose soon began appearing on camera. "A Conversation with Jimmy Carter" that aired on Moyers's TV series U.S.A.: People and Politics won a 1976 Peabody Award. Rose then worked at several networks honing his interview skills until NBC affiliate KXAS-TV in Dallas–Fort Worth hired him as program manager and provided the late-night time slot that became known as 'the Charlie Rose show.'
Rose worked for CBS News (1984–1990) as the anchor of CBS News Nightwatch, the network's first late-night news broadcast, which often featured Rose doing one-on-one interviews with notable people in a similar format to his later PBS show. The Nightwatch broadcast of Rose's interview with Charles Manson won an Emmy Award in 1987. In 1990, Rose left CBS to serve as anchor of Personalities, a Fox TV-produced syndicated program, but six weeks into production and unhappy with the show's soundbite-driven populist tabloid-journalism approach to stories, Rose left. On September 30, 1991, Charlie Rose premiered on PBS station Thirteen/WNET and has been nationally syndicated on PBS since January 1993. In 1994, Rose moved the show to a studio owned by Bloomberg Television, which allowed for high-definition video via satellite-remote interviews.
The New York Times reported that Rose encouraged a discussion between the leaders of NBC and Fox that led eventually to a mutual reduction in ad hominem attacks between Keith Olbermann and Bill O'Reilly on their respective news programs.
Rose received an honorary Doctorate's Degree from SUNY Oswego on October 16, 2014, during the college's annual Lewis B. O'Donnell Media Summit, for his contributions in the broadcast, media and television industries.
Rose's 12-year marriage to Mary Rose (née King) ended in divorce in 1980. Mary is the sister-in-law of the former Morgan Stanley Chairman John J. Mack. Since 1993, Rose's companion has been socialite and chairman of the New York City Planning Commission and director of the city department of planning Amanda Burden, a stepdaughter of CBS founder William S. Paley.
On March 29, 2006, after experiencing shortness of breath in Syria, Rose was flown to Paris and underwent surgery for mitral valve repair in the Georges-Pompidou European Hospital. His surgery was performed under the supervision of Alain F. Carpentier, a pioneer of the procedure. Rose returned to the air on June 12, 2006, with Bill Moyers and Yvette Vega (the show's executive producer), to discuss his surgery and recuperation.
In a 2009 Fresh Dialogues interview, Rose described his life as "great and glorious." He added, "I get up every morning with a new adventure. The adventure is fueled by interesting people. I get a chance to control my own destiny. I do something that is immediately either appreciated or not. I get feedback."