Rahul injured his right hamstring while running between the wickets during the first Test in Kanpur.
"BCCI medical team has confirmed that team India opener, KL Rahul has an injury in his right hamstring and he is ruled out for the remainder of the Paytm Test series against New Zealand," BCCI Secretary Ajay Shirke said in a release.
"The All-India Senior Selection Committee has named Gautam Gambhir as Rahul's replacement for the next two Test matches," he added.
The BCCI also named a replacement for fast bowler Ishant Sharma, who missed the first Test with chikungunya. As he is yet to recover fully, Ishant will also miss the second Test.
Uncapped Haryana off-spinner Jayant Yadav will replace Ishant in the squad for the second Test to be played in Kolkata from September 30.
"Medium pacer Ishant Sharma is recovering and won't be available for the second Test of the series to played in Kolkata," the release said.
"BCCI medical Team will continue to monitor his progress. Off spinner Jayant Yadav will replace Ishant for the second Test."
In a press release, the BCCI said that it would "continue to monitor Ishant's progress".
Gambhir was rewarded for showing good form in the recently concluded Duleep Trophy. He scored 356 runs in the tournament from five innings at 71.20. He scored four half-century and was twice dismissed in the 90s.
The 34-year-old Delhi lad played his last Test in August 2014, on India's tour to England. He aggregated 25 runs on that tour, from four innings. It was a comeback series for Gambhir as he was also dropped in 2012.