India now gives platforms three hours to remove unlawful content once ordered, and asks the largest to check uploads for synthetic media before they go live. Under the IT Amendment Rules 2026, every intermediary must act on a court or government order within three hours, or two for non-consensual intimate or morphed imagery. Services that let users create or share synthetic media must label it and, where feasible, embed provenance metadata; the largest platforms must also verify users’ declarations about uploads. IndiaAI is also funding detectors, such as Saakshya from IIT Jodhpur and IIT Madras, and IIT Kharagpur’s real-time voice deepfake detector.
So can machines tell real from fake? Sometimes. The danger is a machine that won’t admit when it can’t.
Consider Raghunath, a hypothetical 67-year-old retired bank clerk in Patna. His family WhatsApp group receives a clip of his grandson, bandaged in a hospital bed, pleading in Bhojpuri for Rs 40,000. The face and room are real, lifted from the boy’s Instagram, but ten seconds of audio are in a cloned voice. Raghunath runs it through a free detector app, which says: “Real, 92%.”
I build AI systems that reason over text and video, and study how to catch errors that look right. Raghunath’s clip fails every test we know is hard.
Why The Machine Got It Wrong
The fake carried no label:The rules bind platforms and compliant tools; a scammer using an open-source cloner labels nothing, and forwarding often strips metadata anyway. Worse, research on warning labels finds an “implied truth” effect: once some content is labelled, the rest looks more credible.
The fake was out of distribution: Detectors learn from lab data. Researchers who tested open-source detectors on deepfakes circulating online found their performance roughly halved, with many barely better than chance, and blamed academic datasets that no longer resemble today’s fakes. Even that real-world test set was mostly English. Compression widens the gap: lab studies have long shown detection accuracy falls on compressed video. A thrice-forwarded Bhojpuri clip is far from anything detectors were trained on.
The fake was partial: Judging a whole file is easier than locating the problem inside it. InHard2Verify, my colleagues and I tested 29 AI verifiers against expert step-by-step grading of maths solutions. By balanced accuracy, GPT-5 scored about 90% at judging whole solutions but 71% at finding the first wrong step when asked directly (77% when prompted step by step). Weaker verifiers marked almost every step correct. A detector scoring a whole clip can be similarly generous to ten bad seconds inside ninety good ones.
The closest fakes are the hardest: InMedHallu, our benchmark for medical hallucinations, errors closest to the truth were hardest to catch: without supplied knowledge, GPT-4o’s F1 on the hardest cases was just 0.625. A real face with a cloned voice is the audio-visual equivalent. InTeaching with Lies, we trained hallucination detectors on synthetic falsehoods ordered from easy to hard, pushing them past obvious tells; deepfake detectors need such curricula, refreshed as generators change.
The verdict was too sure: “Real, 92%” sounds like evidence. It is a guess with a decimal point. When Raghunath asks a chatbot instead, he gets a fluent answer either way. FaithEval, which I contributed to, tests whether models stick to the context in front of them; across 4,900 problems, even top models often failed, and GPT-4o fell from 96.3% on its own knowledge to 47.5% when a passage contradicted it. One task checks whether a model admits the context lacks the answer; that remains hard.
Policing Can Misfire Too
Errors cut both ways:The rules ask platforms to use automated tools to verify users’ declarations and to block unlawful synthetic content before publication, so detectors will make many calls. One that wrongly flags genuine footage, satire or a whistleblower’s recording blocks real speech, and the rules give users whose uploads are blocked no notice or way to challenge it. Detectors shrunk to run at platform scale also need fairness audits. At Microsoft Research India, we found that compressing language models canworsen fairness, depending on model and method.
What Would Work
None of this needs a breakthrough, just systems designed to admit doubt:
- Provenance first, detection second: Phones and cameras can attach content credentials at capture. The rules already ask for metadata; messaging apps should preserve it on forward, so authentic media can prove itself.
- Three answers, not two: Detectors should say real, synthetic or can’t tell, and “can’t tell” should be shown prominently. In MedHallu, adding a “not sure” option along with domain knowledge improved precision and F1 by up to 38% relative to baselines.
- Point to the problem:A useful detector marks which seconds or which region looks altered, so a person can check it.
- Test where Indians are:IndiaAI’s detection projects should be evaluated on a shared public benchmark of Indian-language, code-mixed and WhatsApp-compressed media, with missed-fake and false-alarm rates published.
- Keep a human in the loop: Where a detector score triggers blocking or a label, require human review, notice to the user and a quick appeal.
- Keep a second channel: For Raghunath, the best detector is a phone call to his grandson’s known number, or a family code word. Awareness campaigns should teach that first.
Machines can help police deepfakes, but only if they are honest about uncertainty. For nearly 1.5 billion Indians, a detector that says “I’m not sure” is not failing. It is the one telling the truth.
(The author is a Research Scientist at Salesforce AI Research, in Palo Alto, California, US)
Disclaimer: The opinions, beliefs, and views expressed by the various authors and forum participants on this website are personal and do not reflect the opinions, beliefs, and views of ABP Network Pvt. Ltd.