Those joining the Congress in not attending the meeting include NDA ally Shiv Sena as well as the Samajwadi Party (SP), the Bahujan Samaj Party (BSP), the DMK, the Telugu Desam Party (TDP) and the Trinamool Congress.
The Left parties, represented by CPI(M) general secretary Sitaram Yechury and CPI's D Raja, attended the meeting but are opposed to the idea.
"We will attend and oppose the 'one nation, one election' issue," a senior Left leader said.
Holding simultaneous polls to Parliament and state Assemblies is fundamentally anti-federal and anti-democratic and thus against the Constitution, Yechury said in a note.
The meeting was held in the library building of Parliament.
The Congress took the decision of staying away from the meeting after consultations with other parties on the issue. The party had convened a meeting of all opposition leaders on Wednesday morning to discuss the issue but it was cancelled in view of its president Rahul Gandhi's birthday.
BSP chief Mayawati tweeted saying she would have attended the all-party meeting if it was on electronic voting machines (EVMs).
According to sources, Kejriwal did not attend the meeting. The Aam Aadmi Party would be represented by party member Raghav Chadha, they said.
The Telangana Rashtra Samithi was represented at the meeting by its working president K T Rama Rao, the son of K Chandrashekar Rao.
Banerjee on Tuesday excused herself from the meeting while asking the Centre to prepare a white paper on "one nation, one election", instead of doing it "hurriedly". As per reports, Kejriwal would not be attending the meeting and the AAP would be represented by party member Raghav Chadha, who was fielded from South Delhi constituency in the recently held Lok Sabha polls.
Modi had invited the heads of all political parties which have at least one member either in the Lok Sabha or Rajya Sabha for the meeting on Wednesday to discuss several issues, including the "one nation, one election" idea, celebration of 75 years of Independence in 2022 and the 150th birth anniversary of Mahatma Gandhi this year.
With 303 MPs, the ruling Bharatiya Janata Party (BJP) at the Centre is the largest party in the Lok Sabha, followed by the Congress (52) and the DMK (23).
In August last year, the Law Commission had recommended holding simultaneous elections to the Lok Sabha and state assemblies to save public money. The Commission, in draft recommendations, said simultaneous polls would help the government of the day focus on "developmental activities rather than electioneering".
The draft, submitted to the Law Ministry, had recommended "holding of simultaneous elections to House of the People (Lok Sabha) and the State Legislative Assemblies (except the State of Jammu and Kashmir)".
It, however, cautioned that "holding simultaneous elections is not possible within the existing framework of the Constitution".
The Centre has been toying with the idea. "The Prime Minister called for widespread debate and consultations on simultaneous elections for Lok Sabha and Vidhan Sabhas, keeping in view various aspects such as the resulting financial savings and consequent better utilisation of resources," an official release said last week after Modi addressed a NITI Aayog meeting here.
The government think-tank, NITI Aayog, had last year suggested synchronised two-phase Lok Sabha and Assembly polls from 2024 to ensure minimum campaign-mode disruption to governance.
WATCH: Reason Behind Mamata Skipping Modi's 'One Nation, One Election' Meeting