"What has happened to the Congress party? The 'naamdar' (dynast, referring to Rahul Gandhi) is tacitly supporting it and therefore, its leaders are dragging the name of my father, who passed away 30 years ago," Modi said, addressing a rally here in poll-bound Madhya Pradesh.
"The Congress is running out of steam in this election, so they have started dragging my parents into politics.
"No Congressman would make such a statement against me or my family without the consent of the party president. It is the 'naamdar' who is making his leaders speak against me," he added.
"My mother sits in her house, performing religious rites, confining herself to a room, and has never visited Madhya Pradesh. She does not even know the 'R' of 'rajneeti' (politics)," Modi said.
"Yesterday, they took the name of my mother and now my father. The Congress chief is behind this...," he alleged.
"Hundred generations of my family have no links with politics. We lived in a small village....lived like the poor.
"And the naamdar says, Modiji is also speaking against my family'," the prime minister said.
"Naamdar, I do not talk about your family members. I talk about former prime ministers, former Congress leaders. I speak against former Congress leaders, seeking an account of their work.
"Yes, I have to take their names as they were in politics, unlike my family members, who sit quietly in their home and do their job," he added.
"You throw questions at Modi, (then) just as Modi is accountable, your family too is answerable," the prime minister said.
Calling the Congress a "burden on the nation", he also justified his own tirade against the Nehru-Gandhi family, pointing out that unlike his parents, they occupied top positions in politics and the government.
The 230-member Madhya Pradesh Assembly will go to the polls on November 28 and the results will be announced on December 11.
(With inputs from PTI)