🔍 Read the full analysis: Z.ai’s Legal AI Passes Roughly 20% Of Tasks And Still Ranks Highly on ThorstenMeyerAI.com
Get the little things that make your day delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Z.ai’s GLM-5.3-FlashX-Legal recorded a 19.17% all-criteria task pass rate on Vals AI’s Harvey Legal Agent Benchmark leaderboard, ranking seventh of 81 models in the archived table. The same run scored 92.5% on individual criteria, showing how the benchmark’s all-or-nothing task rule separates partial coverage from fully completed work.
Z.ai’s GLM-5.3-FlashX-Legal passed every required criterion on 19.17% of tasks in an archived Vals AI leaderboard for Harvey’s Legal Agent Benchmark, ranking seventh among 81 models. The score measures complete task success under an all-or-nothing grading rule; it does not mean the model met only 19.17% of all individual requirements.
Vals AI’s archived leaderboard, last updated October 18, 2026, lists the legal model at $0.15 per test. It records a 92.5% pass rate across individual criteria for the same run. That larger figure reflects partial success on tasks that may still fail the all-criteria test if even one required item is missed.
Z.ai says the result represents a 12.5 percentage-point improvement over the base GLM-5.3-Flash model. The archived table supports that comparison: the legal model scored 19.17%, while the base model scored 6.67%. Z.ai also described the result as state-of-the-art among open-source models in the evaluation, but the material provided does not identify the comparison set, and it reports no published weights for the legal model.
Z.ai’s documentation describes GLM-5.3-FlashX-Legal as optimized for legal research, document understanding, contract review and drafting. The company says it was adapted through legal-domain training and reinforcement learning across tens of thousands of simulated legal-workflow environments. Its October 13 blog post opened a beta by application and said the model was built on GLM-5.3-Flash. The documentation recommends using it as an agent connected to services for laws, cases and company data.
What the Score Means for Legal Work
The result highlights a central challenge in applying AI to legal work: partial accuracy may not be enough when a deliverable must include every material fact, conclusion, citation, deadline or recommendation. A model can satisfy most rubric items and still fail the benchmark’s complete-task standard. The gap between the 92.5% individual-criterion score and the 19.17% all-criteria score makes that distinction visible.
For law firms and legal departments, the ranking is a measure of performance on a defined test, not proof that the system can safely handle client matters without review. The benchmark’s task pass rate does not establish reliability across every jurisdiction, practice area or live case. Procurement decisions also involve questions beyond benchmark results, including how client data is processed, what contractual and regulatory protections apply, and who verifies the model’s citations and work product before use.
The listed test cost is low compared with several higher-ranked entries in the same archived table, but cost per test is only one benchmark measure. It does not establish the expense of a real deployment, including human review, data integrations and correction of errors. A strong leaderboard position can be informative while leaving those operational questions open.
Products Worth Considering
Levoit Vital 200S-P Smart Air Purifier for Large Rooms up to 1875 ft²
Philips Sonicare ProtectiveClean 5300 Rechargeable Electric Toothbrush, Black, with 3 Brush Heads
Fitbit Sense Health & Fitness Smartwatch (White/Gold, International Version)
As an affiliate, we earn on qualifying purchases.
How Harvey Grades Legal Tasks
Harvey released its Legal Agent Benchmark as open source on May 6, 2026. It contains 1,250 tasks across 24 legal practice areas. Each task pairs an instruction with a client matter and asks an agent to produce work for review. More than 75,000 expert-written rubric criteria cover items such as facts, conclusions, citations, severity ratings, recommendations, deadlines, dollar amounts and formatting.
Under Harvey’s all-pass grading rule, a task counts as complete only when every criterion passes. The public leaderboard uses a private holdout set of 120 tasks, according to Vals AI, with internet access disabled. Two language-model judges grade each submission, and the task-pass score averages their results. This makes the reported 19.17% a score from a specific evaluation setup, rather than a general estimate of legal accuracy.
Harvey’s own earlier baseline, published May 26, reported that the best overall all-pass rate among the frontier models it tested was 7.1% and said models completed fewer than 10% of tasks end-to-end in aggregate. Those figures come from a different run and are not directly comparable with the Vals leaderboard result, but they show that complete-task performance has been a demanding benchmark target.
“On Harvey LAB, a benchmark for legal agents, GLM’s legal capabilities achieve an all-criteria pass rate of 19.2%, an improvement of 12.5 percentage points over the base model, achieving state-of-the-art performance among open-source models in this evaluation.”
— Z.ai, in its developer documentation
Limits of the Reported Result
The leaderboard provides a benchmark score, not a deployment guarantee. It remains unclear from the material provided how performance would vary across live client matters, different legal systems, or workflows that use other tools and data. Z.ai recommends connecting the model to external services, but the details of those integrations and the safeguards applied to client information are not specified here.
The source report describes a four-task spot check in which the legal model cited rulings the reviewer could not find. That small check is not a benchmark and cannot establish a general error rate. It does, however, reinforce the need to verify legal citations before relying on generated work. The material also says two Claude models in that spot check cited a real ruling for the opposite of its holding, while OpenAI and Claude models cited no nonexistent rulings in that sample.
Z.ai’s pages, according to the report, describe outputs as drafts for legal professionals but do not carry a “not legal advice” notice. The source material does not establish how the company handles every jurisdiction’s professional or data-protection requirements, or what review process beta users must apply. Those points should not be inferred from the benchmark score.
What to Watch in the Beta
Z.ai’s October 13 announcement said access to the model was opening through a beta by application. The next useful evidence will be more detail on access, deployment options, data handling and the review controls expected of users. The documentation also lists prices of $0.37 per million input tokens and $1.25 per million output tokens, though actual costs depend on usage and any connected services.
Further leaderboard updates could show whether the model’s position changes as additional systems are tested or the evaluation is revised. For legal teams considering a trial, the immediate practical questions are what kinds of matters the beta supports, how its outputs are checked, and whether client data is processed under terms acceptable to the firm. Until those details and broader real-world results are available, the reported ranking is best treated as one performance signal, not a substitute for professional review.
Key Questions
What does the 19.17% score measure?
It is the share of tasks on the archived leaderboard for which every required rubric criterion passed. Missing even one criterion makes a task incomplete under Harvey’s all-pass rule.
Why is the individual-criteria score 92.5%?
The 92.5% figure counts individual rubric criteria, rather than fully completed tasks. A model can meet most criteria across a task while missing one or more, causing the whole task to fail the all-pass measure.
How did the legal model rank?
Vals AI’s archived leaderboard lists GLM-5.3-FlashX-Legal seventh among 81 models, with a 19.17% task pass rate and a listed cost of $0.15 per test.
Does the benchmark show the model is ready to give legal advice?
No. The score reports performance on a specific benchmark and holdout set. It does not establish that the model can replace legal professionals or work safely without checking its facts, citations and recommendations.
What remains unknown about access and data handling?
Z.ai announced a beta by application, but the source material does not detail beta availability, client-data processing terms or required review procedures. Firms would need to assess those issues before using the system with sensitive matters.
Source: ThorstenMeyerAI.com
