The two leaders discussed broad range of topics of mutual interest and Modi said that he was looking forward to visit of PM Abe to India later this year for the Annual Summit.
They held discussions on the global economy, issues of fugitive economic offenders and disaster management.
PM Modi also announced that the Indian President will participate in the coronation ceremony of Emperor Naruhito in October.
It was the first meeting between the two leaders since the start of Japan's Reiwa era and Modi's re-election after the general polls.
Modi thanked Abe for his warm welcome to him and the Indian delegation visiting Japan for the G20 Summit. He also appreciated Japan's leadership as the G20 chairman.
Later, briefing media, Foreign Secretary Vijay Gokhale said it was a "warm" meeting the two prime ministers who are "old friends" and they had a "very constructive and detailed discussion" on bilateral relationship.
"Abe initially began discussion with his expectation from the G20 Summit. They spoke about the importance of focussing on global economy," Gokhale said.
Abe also underlined the need to find appropriate measures to check the global trade issues and data flows and underlined that the G20 should also come up with a constructive message on the issue of climate change.
"These are the areas where Abe hoped that he would get the understanding and support of the G20 members, including that of India," Gokhale said.
"Abe emphasised the importance of preparing for this summit properly including through ministerial meetings, through various exchanges in infrastructure, defence, outer space, digital economy and start-ups," the foreign secretary said.
The two leaders also had a brief discussion on the Mumbai-Ahmedabad High Speed Rail Corridor and on a convention centre being built in Varanasi, Prime Minister Modi's constituency. They agreed on the delivery of both the projects on time.
Prime Minister Modi said he is looking forward to seeing Abe again at the Eastern Economic forum in Vladivostok in early September. Modi will be the chief guest the forum at the invitation of Russian President Vladimir Putin.
Modi arrived in this Japanese city earlier on Thursday. According to the Prime Minister's Office (PMO), Modi "will elaborate on many issues of global importance and present India's viewpoint" during the summit.
On the Modi-Abe meeting, the PMO tweeted: "A friendship characterized by warmth and the promise of a bright future. Prime Ministers @narendramodi and @AbeShinzo hold talks in Osaka, the first such meeting between these leaders since the start of Japan's Reiwa era.
"Many aspects of India-Japan relations were discussed."
During the two days, Modi will meet other world leaders including US President Donald Trump and also hold a trilateral meeting on the sidelines of the summit with Russian and Chinese Presidents Vladimir Putin and Xi Jinping, respectively.
(with inputs from agencies)