🔍 Read the full analysis: 42 AI Setups Answered 15 Legal Questions—Here’s The Score on ThorstenMeyerAI.com
Get the little things that make your day delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Thorsten Meyer AI reports that 42 AI setups answered 15 questions on German, U.S. and EU law, with Claude Opus 5.5 scoring highest at 79 out of 100. The legal-focused GLM-5.3 FlashX Legal scored 8, while the test’s author says citation errors were widespread. The results come from a disclosed, limited spot check and are not legal advice.
Thorsten Meyer AI reported on November 4 that 42 AI setups answered 15 legal questions, with Claude Opus 5.5 scoring 79 out of 100 and legal-focused GLM-5.3 FlashX Legal scoring 8. The results matter to people considering chatbots for legal work, but the test is a limited spot check run by the publisher, who discloses that his site operates the evaluation.
The test, called Lexassay, covers questions involving German, U.S. and EU law. Its 15 tasks include calculating legal deadlines, identifying recent legal changes and court rulings, and spotting questions built on false assumptions. Ten questions are public and five are hidden; the publisher says only scores for the hidden tasks are released.
According to the published results, GPT-6 Astra scored 72 and GPT-6.1 Sol scored 71, behind Claude Opus 5.5. The article says the top model fully met every criterion on only eight of the ten public tasks. Legal-focused GLM-5.3 FlashX Legal received 8 overall and zero on the hidden set; its base model scored 3. The publisher reports that the legal model fully solved none of the 15 tasks and gave no answer on two.
The article also reports 2,644 citations verified as real and 804 found to be invented or not to support the claims attached to them, across all published runs. Under Lexassay’s rules, each invented or misattributed citation costs a quarter of the task’s points. The test also deducts 5% from an answer that lacks a legal-advice disclaimer or a recommendation to consult a lawyer.
Infographic 01 · The leaderboard
Same 15 legal questions, very different results
German, US and EU law, no internet, no tools. Every cited law and ruling checked against the source.
The top · 67 to 79
The middle · 41 to 59
The weak group · 0 to 35
Selection of 14 of 42 configurations. Score 0–100 = share of achievable points, with deductions for wrong citations. Small local models (1–12) not shown.
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Source: Lexassay, release of 4 Nov 2026.
Powered by Thorsten Meyer AI
Legal AI Scores Need Careful Reading
The scores suggest that a model marketed for legal workflows may not perform well on this particular set of questions, while strong overall scores do not remove the need to check individual answers. A legal deadline can have direct consequences: the source article describes a German civil appeal scenario in which a public holiday changes the deadline in Munich but not Berlin. It says one tested model gave Berlin an extra day and another placed the deadline for appeal grounds a month late.
That makes citation accuracy and local legal details as important as a headline score. A chatbot may produce plausible wording while citing a ruling incorrectly or missing a date-sensitive rule. Meyer’s stated recommendation is to use AI as a starting point, verify every cited law and decision against the source, and not rely on a chatbot to calculate a deadline. The article also advises consulting a lawyer when money, deadlines or authorities are involved.
Products Worth Considering
Oral-B iO5 Rechargeable Electric Toothbrush with 5 Cleaning Modes and Travel Case, White
Garmin Forerunner 570 GPS Running Smartwatch, 47mm, Aluminum, Black
Fitbit Sense 2 Smartwatch – Blue Mist / Pale Gold
As an affiliate, we earn on qualifying purchases.
How Lexassay Ran the Test
Lexassay gave each setup the same questions and, according to its method description, ran models without internet access or tools. It grades answers against a prepared answer using three to six weighted criteria and checks citations against official databases. Different effort settings are tested separately, so the 42 setups do not necessarily represent 42 distinct underlying models.
The publisher calls the work a spot check, not a formal benchmark: each model and task receives one answer, meaning small score differences may reflect variation in individual responses. The published article says another evaluation, Vals AI’s legal-agent benchmark, ranks some of the same models differently. That benchmark supplies documents to models, unlike Lexassay’s no-tools test, so the scores measure different tasks and should not be treated as directly interchangeable.
Meyer discloses that Thorsten Meyer AI runs Lexassay and that several Claude agents graded the answers; Claude Opus 5.5, a Claude model, ranked first. He says the method page and public answers, gold answers and citation checks are available for review. The disclosed connection is relevant when interpreting the results, even though publication of the underlying material allows readers to inspect the grading.
“Optimized for professional legal workflows, with a focus on document understanding, legal reasoning, and task execution.”
— Z.ai, as quoted in its developer documentation
Limits Behind the Reported Scores
The reported results do not establish how the models would perform with legal documents, web search, specialist tools or repeated attempts. Lexassay tested one answer per model and task, and the source says a few points of difference may be noise. The full question set is not entirely public: five tasks are hidden, with scores published but not the answers.
The article’s author runs the test and used Claude agents to grade it, while a Claude model placed first. Meyer discloses that conflict and says the materials are public, but the supplied source does not include an independent replication or an external audit of the scores. It also does not provide a complete score breakdown for all 42 setups in the material supplied here. The figures should be read as results from this particular evaluation, not as a general ranking of legal AI performance.
The source excerpt ends during its discussion of GLM-5.3 FlashX Legal and does not include the full response from Z.ai or details of any challenge to the test. It is therefore unclear whether the company has commented on the reported score or whether it disputes the grading.
Independent Checks and Model Updates
The source does not announce a scheduled follow-up test or a response from the model makers. Readers can check the method and published answer materials on Lexassay, according to Meyer, and compare the results with evaluations that test models using supplied legal documents. Future results would be more informative if they included repeated runs and independent grading, given the spot-check design and the disclosed relationship between the publisher, graders and leading model.
For now, the practical next step is verification rather than reliance: check citations in official sources and consult a qualified lawyer for a case-specific deadline, contract or communication with an authority. The test evaluates model answers; it does not determine legal rights in an individual case.
Key Questions
Which AI setup scored highest?
Claude Opus 5.5 led the reported results with 79 out of 100. It fully met every criterion on eight of the ten public tasks, according to the article.
How did the legal-focused model perform?
GLM-5.3 FlashX Legal scored 8 out of 100 and received zero on the hidden set. The publisher says it fully solved none of the 15 tasks and did not answer two of them.
Does this prove which AI is best for legal work?
No. Lexassay’s publisher describes it as a spot check rather than a strict benchmark, with one answer per model and task. It also tests models without internet access or tools, and a separate benchmark using supplied documents produced different rankings.
Can I use a chatbot to calculate a legal deadline?
The article advises readers not to rely on a chatbot for deadlines. It recommends checking cited law and rulings against official sources and asking a lawyer when a deadline, money or an authority is involved.
Source: ThorstenMeyerAI.com
