The Union minister was addressing the gathering at the 'Akhil Bhartiya Mali Samaj Mahaadiveshan', organised by the Mahatma Phule Shikshan Sanstha.
Gadkari, a former BJP president, said, "When people fail to get (poll) ticket on the basis of their work (krututva), they play caste card."
Listing a number of prominent Indian politicians who rose above the identity of caste and religion, he said that many leaders of the country made an identity of themselves which remained unaffected by their caste.
"I want to ask - did George Fernandes (a former Union minister and socialist stalwart) belong to any caste? He did not belong to any caste...he was a Christian. Did Indira Gandhi come to power because of caste?
"Did Ashok Gehlot belong to your caste? But he became Chief Minister of Rajasthan when people from other caste groups helped him," he said.
"People had told me that women should get reservation, 'I said yes they should get reservation'. But, I immediately asked them, did Indira Gandhi get reservation. For several years she ruled the country and became popular.
"Similarly, did Vasundhara Raje and Sushma Swaraj (BJP leaders) get reservation?" said Gadkari to buttress his argument that caste was not instrumental in success of these politicians.
Gadkari said, "In politics those who do good work, don't have to ask for votes as the votes come to them naturally."
"Prime Minister Modi never talks about his caste. I really appreciate Modiji. PM Modi has never said he belongs to a backward community," Gadkari said.
(with PTI inputs)