The temple will be hand-carved by Indian temple artisans and assembled in the UAE. The UAE has two Hindu temples which are located in Dubai. Devotees from Abu Dhabi and other emirates have to drive to Dubai for prayers and offerings.
Modi laid the foundation stone for the BAPS Swaminarayan temple via video conferencing from the Dubai Opera House, where he addressed the members of the Indian community.
"I would like to request all those associated with the construction of the temple that the rulers here have shown so much respect for India. They have been proud of India's cultural history. Now it's our responsibility that there is no mistake on our part," he told the gathering, amid applause.
The temple compound will include a visitors' centre, prayer halls, exhibition centre, learning areas, sports area for children and youths, thematic gardens, water features, a food court, a books and gift shop and other facilities.
The UAE is home to around 3.3 million expatriate Indians.
Modi asked people associated with the construction of the temple and also the worshippers not to cause any harm to others. "This is my expectations from you," he added.
He also unveiled the model of the temple and thanked the Crown Prince of Abu Dhabi for gracious donation of the land.
"People were surprised when Abu Dhabi's Crown Prince took forward the talks of building a temple in Abu Dhabi. I whole heartedly thank his highness on behalf of all 125 crore Indians," Modi said, amid applause.
Modi said this temple will also become a "medium of India's identity".
"We have grown in a culture where temples are a medium of humanity...Holy place is a catalytic agent of humanity and harmony," he told the gathering.
"I believe this temple will not be only unique in terms of architecture and splendour, but will also give a message of 'Vasudhaiva Kutumbakam' to people across the world," he said.
Temple Committee members had presented the temple literature to Modi and Mohamed bin Zayed Al Nahyan last evening in Abu Dhabi.
"PM @narendramodi witnessed laying of foundation stone for the BAPS Swaminarayan temple on Abu Dhabi - Dubai highway! The first Hindu temple in Abu Dhabi being built on a generous gift of land by the Crown Prince of Abu Dhabi reflects UAE's commitment to tolerance and harmony," Ministry of External Affairs Spokesperson Raveesh Kumar tweeted.
Modi is visiting the United Arab Emirates for the second time after his 2015 trip to the Gulf country.
The temple will come up on 55,000 square metres of land. The structure will be hand-carved by Indian temple artisans and assembled in the UAE. It will be completed by 2020, and open to people of all religious backgrounds.
It will be the first traditional Hindu stone temple in the Middle East, said a spokesperson from the Bochasanwasi Shri Akshar Purushottam Swaminarayan Sanstha (BAPS), a socio- spiritual Hindu organisation set up in 1907 that runs more than 1,100 temples and cultural compounds around the world.
The temple will incorporate all aspects and features of a traditional Hindu temple as part of a fully functional, social, cultural and spiritual complex.
It will replicate the BAPS temple in New Delhi and the one under construction in New Jersey.
(With inputs from agencies)