Harvey Unveils Initial Results of Legal AI Benchmark LAB

Harvey has released the first results from its Legal Agent Benchmark (LAB), an open-source framework designed to evaluate AI agents on complex, long-horizon legal tasks. The initial findings underscore significant limitations in current-generation AI models. Despite rapid advancements, frontier models completed less than 10% of LAB tasks end-to-end under a strict all-or-nothing evaluation standard. LAB evaluates AI agents across over 1,200 tasks spanning 24 legal practice areas. Each task mirrors real-world law firm workflows, requiring AI models to produce review-ready legal work products graded against 75,000 expert-created rubric criteria. Harvey's "all-pass" scoring system demands perfection—every rubric criterion must be satisfied for a task to pass. Among the evaluated models, Claude Opus 4.7 led with a 7.1% success rate, followed by Sonnet 4.6 at 5.4%, Opus 4.6 at 4.2%, GPT-5.5 at 2.1%, and Gemini 3.5 Flash at just 0.8%. The findings also revealed uneven competence across practice areas, with models displaying "jagged intelligence." No single model dominated across all categories, reinforcing the need for multi-model strategies in AI deployments. Another major hurdle is operational efficiency. The best-performing model, Opus 4.7, costs approximately $50.90 per task and has a latency of 22 minutes. Faster alternatives like Gemini 3.5 Flash offer lower latency but at the expense of accuracy. Harvey's study also analyzed agent behavior, identifying key patterns that improve task performance. The most effective agents demonstrated behaviors akin to those of skilled human associates: thorough research before drafting, post-draft validation, and iterative revisions. The benchmark's next phases will focus on expanding its task library, improving cost-efficiency, and fostering collaboration with AI labs to refine model performance.

(Source：Blockchain News)