Aziz and Indian national security adviser Ajit Doval had met briefly on Saturday evening.
While India - which has asked Pakistan to stop cross-border terrorism for meaningful talks - insisted that the conversation was brief and took place during a short stroll, Pakistan suggested it was more meaningful.
Late this afternoon, Aziz was stopped from driving to the Golden Temple, which Modi and Afghan President Ashraf Ghani had visited on Saturday. Pakistan officials claimed Aziz was barred from visiting the shrine to pre-empt an opportunity for him to speak to reporters.
The Pakistan high commission in India then began multiple attempts for Aziz to hold a media interaction but all were stymied. The friction culminated in an exchange the country's high commissioner Abdul Basit had with a security person just outside the media centre here, when he too was stopped from entering.
The exchange was recorded by a Pakistani journalist, who shared it with this newspaper.
"I just want to talk to Pakistani journalists," Basit can be heard telling the security person wearing a blue turban. "Nobody can stop me. These are my people."
Indian officials insisted that Aziz's movements were restricted purely because of security concerns. Aziz finally flew out for Islamabad from here at 8pm.
A senior Indian official said the decision to curtail his movements was a result of India's keenness to "avoid anything adverse happening to any of our guests".
Indian officials contended that the decision was influenced by the fact that December 4 was Guru Teg Bahadur's martyrdom day, and a visit by the Pakistani delegation would have inconvenienced other devotees.
Most other visiting delegations were taken to the Golden Temple Saturday.
But Pakistani officials said they were unconvinced by India's explanation, though late in the evening, at a media conference immediately after landing in Islamabad, Aziz tried to downplay the treatment meted out to him, while expressing concerns over his access to the Pakistani media.
"The way the media was treated was absolutely not right," Aziz said, adding that he hadn't tried much to reach out to Indian journalists but was keen to speak to Pakistani journalists who had traveled with him. "They had come all the way, and I thought I would speak to them there."
After Aziz - who was staying at the summit venue, the Radisson Blu hotel - was barred from visiting the Golden Temple, the Pakistan high commission tried hosting a media briefing at another venue: the Holiday Inn about 7km away.
Pakistani journalists who were told about Aziz's briefing rushed there, only to be told that the adviser was not being allowed to leave his hotel for the interaction. The journalists were then advised by the Pakistan high commission to go to the Radisson Blu.
But there, they were not allowed to enter the hotel - and Aziz wasn't allowed to go to the entrance to address them.
At that point, Basit walked to the media centre, about 800 metres outside the Radisson Blu.
The video recording shows Basit arguing with a security person at the entrance of the media centre, when he was forbidden from entering and speaking with reporters.