Kozma was drafted by the Cardinals in the first round of the 2007 amateur draft out of Owasso, and began his professional career that year.
In his first professional season, Kozma played for three minor league teams - the Johnson City Cardinals (30 games), GCL Cardinals (four games) and Batavia Muckdogs (eight games). Overall, he hit .233 with two home runs and 11 RBI in 42 games that season.
In 2009, he split the season between Palm Beach (18 games) and the Springfield Cardinals (113 games), hitting .231 with six home runs and 45 RBI in 131 games. He played for Springfield in 2010, hitting .243 with 13 home runs and 72 RBI in 132 games. He also stole 13 bases in 15 tries.
On May 18, 2011, Kozma was called up from the AAA Memphis Redbirds, with whom he hit .214 in 112 games, to the major league club to replace injured middle infielder Nick Punto. He made his major league debut that night, and hit a double with an RBI in his first at-bat when pinch-hitting in the 5th at Busch Stadium. Kozma was part of the Cardinals postseason roster when the team won the 2011 World Series over the Texas Rangers.
Kozma was optioned to Triple-A to begin the 2012 season. He was recalled to the Cardinals on August 31, 2012 when Rafael Furcal was placed on the disabled list. Kozma singled to give the Cardinals the deciding runs in the winner-take-all fifth game of the 2012 National League Division Series over the Washington Nationals. During the 2012 NLDS, Kozma hit 1 home run (his first in a postseason) with 5 RBI. Kozma batted .227 during the 2012 NLCS and had 1 stolen base. Despite the Cardinals having an advantage 3 games to 1 in the series, they lost to the San Francisco Giants in the 7th and decisive game.
In Game 1 of the 2013 World Series against the Boston Red Sox, Kozma was involved in a controversial error/blown call that was overturned by the umpiring crew. Kozma could not handle a ball tossed to him that would have commenced an inning ending double play. Red Sox manager John Farrell stormed out of the dugout, confronted umpire Dana DeMuth and requested that the umpire crew get together and make a group decision on the ruling. The umpires ultimately ruled that Kozma never had control of the ball, and the inning continued. Boston went on to score 3 runs that inning, and eventually went on to win the 2013 World Series.