Addressing an election rally at Sanand, Modi said the meeting at Aiyar's residence was also attended by former Indian Vice President Hamid Ansari and former Prime Minister Manmohan Singh.
Continuing his attack on the Congress, Modi said: "There was a meeting convened at Aiyar's residence where Pakistan's High Commissioner, Pakistan's Foreign Minister, Ansari and Manmohan Singh were present. The meeting lasted three hours.
"The next day Mani Shankar said I am a 'neech'," said the Prime Minister.
Modi went on: "I ask you, brothers and sisters, why did they have to have a meeting with Pakistani officials. Pakistan is our enemy. You (Congress) meet them and the next day call me 'neech'.
"Isn't this a thing to worry about? Does it not raise doubts? There is something dubious here, isn't it? You should have ensured the presence of government officials at the meeting. Why are you holding meetings with our enemy neighbour in secrecy?"
"Isn't calling me 'neech' insulting to Gujarat? Isn't it insulting to the country? Insulting to the government whom the people voted to power?"
The Prime Minister added that a former Pakistan Army general, Arshad Rafiq, has announced that he favoured Congress MP Ahmed Patel as the Gujarat Chief Minister.
"Why? You are in Pakistan. Why are you interested in our country's affairs?
"Brothers and sisters, isn't this alarming? All this and the meeting at Aiyar's residence. What is all this? My friends, you should give me a promise that you will root out the Congress and give a thumping victory to the BJP at every booth in Ahmedabad."