Inside Legal AI: What's Actually Under the Hood

The article explores the underlying technology powering legal AI products, revealing that many are built upon the same foundation models from providers like OpenAI, Anthropic, and Google – the same engines driving popular consumer AI tools like ChatGPT. While these models are capable, the value added by legal tech companies lies in their custom interfaces, security infrastructure, and, crucially, how they retrieve and structure information. The article details the process of Retrieval-Augmented Generation (RAG), where documents are converted into vector embeddings to find relevant passages for the AI, and highlights the limitations of standard RAG implementations. A newer approach, 'agentic retrieval,' iteratively plans and executes retrieval steps, mimicking a human investigator's process for more accurate results in complex tasks. The article also addresses the 'context problem,' where performance degrades as the amount of information fed to the model increases, advocating for careful context engineering. Ultimately, the piece suggests that for simple tasks, the difference between legal AI products and direct use of foundational models is minimal, but for complex work, architectural choices regarding retrieval and context management are significant, and practitioner supervision remains essential.

(Source：National Law Review)