Legal AI tools used by law firms have a propensity to hallucinate—can anything be done?
Law firms must invest in new verification systems to manage AI hallucinations, as current oversight models are insufficient for AI-generated work.Summary
A recent incident involving Sullivan & Cromwell, which apologized for AI-hallucinated citations in court filings, highlights a growing crisis in professional service firms. The traditional model of leverage, where partners oversee associates, is breaking down because AI tools produce vast amounts of material but lack the uncertainty signals that human associates provide, making verification difficult. Researchers have found that leading legal AI products hallucinate between 17% and 33% of the time, and vendors' claims of 'hallucination-free' systems have been disproven by testing. This creates a principal-agent problem where partners may prioritize short-term productivity gains over long-term risk management. To capture the economic benefits of AI, firms must proactively build a new verification function that operates at a larger scale, likely requiring changes in staffing, training, and the adoption of complementary organizational innovations.
(Source：Livemint)