Gandhi has put forth many conditions for the visit, including meeting mainstream leaders under detention, the governor said in a statement.
Hitting back at the governor for "taking a U-turn" on his offer to visit the state, the Congress said he should stand by his word.
"The Governor is taking a U-turn. He made an offer that anyone can visit the valley to judge for themselves. Shri @RahulGandhi has taken him up on the offer. He should stand by his word & allow a multi-party delegation to visit Jammu and Kashmir unhindered," the Congress said on its official Twitter handle.
Malik said he had never invited the Congress leader with "so many pre-conditions" and has referred the matter to the local police and administration for further examination.
"Rahul Gandhi is politicising the matter by seeking to bring a delegation of opposition leaders to create further unrest and problems for the common people," he said in the statement.
On Tuesday, Gandhi accepted Malik's 'invitation' to visit the state but said he didn't need an aircraft and that he and other opposition leaders would visit. The Congress leader in a tweet also urged the governor to allow them the freedom to meet the people and soldiers.
On the Congress leader's comments on violence in Kashmir, Malik said, "Rahul Gandhi is responding to fake news possibly spread from across the border about the situation in Kashmir, which is peaceful with negligible incidents."
He can check for himself from various Indian channels that have reported the correct position in the Kashmir Valley, the governor said.
"He (Gandhi) can also check the detailed submissions made by the government in the Supreme Court today which heard a case on this matter and left it to government," Malik said.
ALSO WATCH: Priyanka Calls Manner Of Art. 370 Revocation 'Unconstitutional'