In 2009, he split the season between Palm Beach (18 games) and the Springfield Cardinals (113 games), hitting .231 with six home runs and 45 RBI in 131 games. He played for Springfield in 2010, hitting .243 with 13 home runs and 72 RBI in 132 games. He also stole 13 bases in 15 tries.
On May 18, 2011, Kozma was called up from the AAA Memphis Redbirds, with whom he hit .214 in 112 games, to the major league club to replace injured middle infielder Nick Punto. He made his major league debut that night, and hit a double with an RBI in his first at-bat when pinch-hitting in the 5th at Busch Stadium. Kozma was part of the Cardinals postseason roster when the team won the 2011 World Series over the Texas Rangers.
Kozma was optioned to Triple-A to begin the 2012 season. He was recalled to the Cardinals on August 31, 2012 when Rafael Furcal was placed on the disabled list. Kozma singled to give the Cardinals the deciding runs in the winner-take-all fifth game of the 2012 National League Division Series (NLDS) over the Washington Nationals. During the NLDS, he hit a home run – his first in an MLB postseason – and drove in five runners. Kozma batted .227 during the NLCS against the San Francisco Giants and added a stolen base. Despite attaining a 3-games-to-1 advantage in the series, the Cardinals lost the Series in the 7th and deciding game.
Kozma became the Cardinals' primary shortstop in 2013 and finished with a .217 batting average, 20 doubles, a home run and three stolen bases in 143 games. They secured the best record in the National League, thus making the playoffs. He provided a key defensive play in the sixth inning of Game 3 of the NLDS against the Los Angeles Dodgers. Dodgers batter Juan Uribe hit a rapid ground ball in the hole, which Kozma backhanded and relayed to second baseman Matt Carpenter, tipping off an inning ending double play. The Cardinals won the game, 4–2.
The Cardinals won the NLCS to advance to the World Series. In Game 1 of the World Series against the Boston Red Sox, Kozma was involved in a controversial error/blown call that was overturned by the umpiring crew. He mishandled and dropped a ball tossed to him that would have commenced an inning-ending double play. Red Sox managerJohn Farrell leapt of the dugout, confronted umpire Dana DeMuth and requested that the umpire crew get together and make a group decision on the ruling. The umpires ultimately ruled that Kozma never had control of the ball, and the inning continued. Boston went on to score three runs that inning, and eventually went on to win the championship in six games.
The following offseason, the Cardinals signed longtime American League player Jhonny Peralta as a free agent to upgrade their offense at shortstop, thus supplanting Kozma from his short-lived status as the starter. The Cardinals signed Mark Ellis, another free agent, and kept rookie second baseman Kolten Wong on the MLB roster for much of the season. Those three moves relegated Kozma to spending most of 2014 playing for Memphis, where he hit .248 with eight home runs and 59 RBI. The Cardinals recalled him when MLB rosters expand in September, and, in fourteen games, hit .304 with three doubles. The club won the National League Central division, assuring a playoff berth for the fourth time in his four seasons with the club. He was a last-minute addition to the postseason roster for the NLDS against the Dodgers.
The following spring training, Kozma resumed working out at all infield positions, and added time as a catcher. On June 25, 2015, against the Miami Marlins, he scored on a double from Kolten Wong. He had three hits and reached base in four plate appearances, snapping an 0–21 streak that dated back to May 19. Kozma totaled 99 at bats for the season, batting .152 with a .236 on-base percentage and no extra base hits. The Cardinals removed him from the 40-man roster after the season and reassigned him to Memphis, after which the New York Yankees signed him to a minor-league contract.