The peril of Gen AI in court: Why human verification is non-negotiable

The Supreme Court of India has warned against the uncritical use of Generative AI in judicial processes, highlighting the potential for “hallucinations” – the generation of false or nonexistent case law. This could lead to skewed inferences and judicial misconduct, as evidenced by recent cases and warnings from High Courts in India and similar issues globally. Experts like Mangari Rajender, a retired district judge, emphasize that judicial decisions must be based on human reasoning and careful consideration, not AI reliance. Sanjeev Kapoor of Khaitan & Co notes that US courts have sanctioned lawyers for submitting unverified AI-generated citations.

While AI can be a valuable assistive tool for tasks like summarizing case law and refining drafts, it should not replace human professional judgment. AI Industry Analyst Kashyap Kompella stresses that AI tools like ChatGPT and Gemini generate text based on predicted word sequences, lacking factual verification and legal understanding. He advocates for purpose-built legal AI platforms trained on curated databases.

Ultimately, the core concern is the uncritical acceptance of AI outputs. The Supreme Court’s ruling reinforces the importance of reasoned analysis, verified precedent, and accountability in judicial decisions, recognizing that a judgment is an exercise of State power grounded in law.

(Source：The Hindu - Business Line)