During questioning, Rathi told the police he shot Bajrangi after an altercation took place between the two in the morning when both were out of their cell for having tea.
When Bajrangi put the pistol on Rathi's chest after a heated exchange, he shot him with the same weapon, police said.
The police have recovered the weapon from a septic tank of the jail. Ten empty shells of a .762 bore weapon and some live cartridges were recovered from inside the district jail.
Bajrangi was kept in a cell with 10 other inmates, including Rathi. The 51-year-old criminal was shot ten times and then his body was dumped into a gutter inside the prison premises.
ALSO READ: Who was Don Munna Bajrangi?
Considering the major loophole in security arrangements, four jail officials, including the jailer and Deputy Jailer, were suspended soon after the incident.
Bajrangi was transferred with full security from Jhansi to Baghpat jail yesterday and was to be produced in a local court today in a case of demanding extortion money from former BSP MLA Lokesh Dixit in 2017.
Uttar Pradesh Chief Minister Yogi Adityanath termed the killing of 51-year-old Bajrangi, who was listed as an accused in 40 criminal cases, including those of murder and extortion, "serious" and ordered a probe.
Bajrangi's wife Seema Singh, who had called a press conference 10 days ago on June 29 to allege that her husband's life was in danger, has demanded a CBI probe.
"I want to tell UP Chief Minister Adityanathji that a conspiracy is being hatched to kill him in a fake encounter," Seema had told reporters in Lucknow on June 29.
ALSO READ: Don Munna Bajrangi shot dead by another inmate in Baghpat Jail
Bajrangi was also named, along with BSP MLA Mukhtar Ansari, in the killing of former BJP legislator Krishnanad Rai in 2005.
"It is divine justice. I am happy," Rai's wife Alka Rai told reporters after news of Bajrangi's killing came in.
Bajrangi had unsuccessfully contested the 2012 Assembly polls from Madiyahu seat in Jaunpur on an Apna Dal ticket.
In March 2016, his brother-in-law Pushpjeet, who looked after his business, was shot dead in Vikas Nagar colony of Lucknow and his close associate Tariq was shot at in Lucknow in December last year.
As a teenager, Bajrangi is believed to have developed an affinity for arms and entered the world of crime at an early age.
The first case against him -- for fighting and possessing illegal arms -- was filed when he was just 17 years old.
He carried out his first major crime in 1984 after he killed a businessman. Bajrangi also murdered BJP leader Ramchandra Singh, police officials said.
Bajrangi, who was also known as mafia-turned-politician Mukhtar Ansari's right hand, had joined Ansari's gang in the 90s, they said.