No. 2-ranked Djokovic had won five of the six previous titles at Melbourne Park and six overall, and had never dropped a set in six previous meetings against Istomin.
It was destined to be a long Thursday afternoon on Rod Laver Arena for Djokovic, though, when he needed 24 serves and a quarter of an hour to hold in the first game, saving six break points.
He lost the 85-minute first set in a tiebreaker, seemed to get the momentum back only for No. 117-ranked Istomin to finish stronger for a 7-6 (8), 5-7, 2-6, 7-6 (5), 6-4 win.
"All the credit to Denis for playing amazing. He deserves to win," Djokovic said. "He was the better player in the clutch moments.
"Obviously, I was not pleased with my performance overall, but I have to congratulate my opponent. Whenever he needed — he came up with a big serve, big play."
Istomin had never beaten a top 10 player at a Grand Slam tournament.
"It is the biggest win for me. It means so much," Istomin said. "Now I feel I can play with these guys, and to be with them on the same level."
His only previous win in 33 matches against a top 10 player was in 2012 at Indian Wells, when he beat No. 5 David Ferrer.
Djokovic had reached the fourth round or better every year since 2007 — the year before he won his first Grand Slam title at Melbourne Park — and held the No. 1 ranking for 122 consecutive weeks until he was overhauled by Andy Murray last November.
"I'm not used to losing Australian Open second round," Djokovic said. "I've done so well here."
The last time he'd failed to advance beyond the second round at a major was at Wimbledon in 2008, when he was seeded third and lost to then No. 75-ranked Marat Safin, a two-time Grand Slam champion.
Rod Laver Arena wasn't at capacity as the match extended into the scheduled night session, and Djokovic's usually vocal Serbian fans were also not a big presence. Much of the crowd, sensing an upset, was behind Istomin in the fifth set, yelling "Come on Denis!"
When Istomin made a backhand winner down the line to break Djokovic to go up 3-2, he let out a roar and the audience stood and roared, too.
Djokovic, meanwhile, looked relatively subdued as the set wore on, hanging his head in between points and walking slowly behind the baseline to prepare to receive serve.
When Djokovic's last service return floated long, the crowd leapt up and cheered as Istomin thrust his arms into the air.
"First of all, I feel sorry for Novak," said Istomin, who thanked his mother and coach, Klaudiya Istomina, in the crowd. "I was playing so good today. I surprised myself, as well."
He will play No. 30 Pablo Carreno Busta in the third round
Third-seeded Milos Raonic shrugged off the effects of a cold to extend his streak of reaching the third round at all seven Australian Opens he's contest with a 6-3, 6-4, 7-6 (4) over Gilles Muller, the 33-year-old Luxembourg player who won his first title at tour level last week in Sydney. He next faces No. 25 Gilles Simon.
Also advancing were Brisbane International winner Grigor Dimitrov, No. 18 Richard Gasquet, U.S. Open semifinalist Gael Monfils, No. 11 David Goffin and David Ferrer, who beat U.S. qualifier Ernesto Escobedo 2-6, 6-4, 6-4, 6-2 to set up a third-round match against No. 13-seeded Roberto Bautista Agut.
Ivo Karolovic followed up his tournament record 84-game first-round win with a straight sets win over wild card Andrew Whittington.
Not long after Djokovic's exit, women's No. 3 Agnieszka Radwanska lost 6-3, 6-2 to Mirjana Lucic-Baroni.
U.S. Open finalist Karolina Pliskova and No. 9 Johanna Konta carried their winning form from warmup tournaments into the third round.
Brisbane champion Pliskova beat Anna Blinkova 6-0, 6-2 in 59 minutes.
Sydney International winner Konta, a semifinalist here last year, beat Naomi Osaka and will next play former No. 1-ranked Caroline Wozniacki.
WTA Finals winner Dominika Cibulkova beat Hsieh Su-wei and No. 14 Elena Vesnina advanced to a third-round encounter against U.S. qualifier Jennifer Brady, who saved five match points before beating Heather Watson.
Murray, a five-time Australian Open runner-up, returned to the practice court to allay concern over his ankle 14 hours tumbling to the court during his second-round win.