🔍 Read the full analysis: Why AI’s Verification Challenge Is An Economics Problem on ThorstenMeyerAI.com
Get the little things that make your day delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A source analysis argues that AI is making it cheaper to produce mathematical, software and legal work without reducing the effort required to establish whether it is reliable. Examples cited include 722 AI-produced mathematical manuscripts and software data showing review delays and limited human oversight. The scale of the effects is uncertain, and several software-data providers sell review tools.
An analysis published this week argues that AI is making work cheaper to produce faster than it is to verify, a mismatch illustrated by a reported release of 722 AI-generated mathematical manuscripts and evidence of growing review pressures in software. The examples point to a possible constraint on how much AI-generated work organisations can safely use: human judgment and accountability.
The analysis says OpenAI posed about 4,000 mathematical problems to a model and produced 722 manuscripts, grouped into 372 families. The average result took about three hours of compute, according to the source. Some results were formally checked in Lean, a proof assistant; OpenAI warned that some results without formal verification could have issues. The source contrasts that output with the careful scrutiny by five leading mathematicians of an earlier result from the programme, described as a counterexample to an old Erdős conjecture.
In software, the analysis cites data from Faros AI showing teams merged 98% more pull requests across periods of low and high AI adoption, while review time rose 91%. It also cites LinearB’s analysis of 8.1 million pull requests across 4,800 organisations: AI-generated changes reportedly waited 4.6 times longer for review to begin and were accepted 32.7% of the time, compared with 84.4% for human-written changes. A peer-reviewed 2026 study, as summarized by the source, found 61% of AI-agent pull requests received no human review before merging or closing.
The source also points to OpenAI’s partnership with contract-software company Ironclad. It says GPT-6 Astra met an average of 55% of evaluation criteria across 11 tasks, an improvement over the prior model. That result indicates progress on the evaluated tasks, but also leaves unanswered how the remaining criteria affect whether the work is usable in real contracting workflows.
The referee shortage: AI made doing cheap and checking expensive
OpenAI’s model produced a maths result in about three hours of compute. Verifying one earlier result took five of the world’s leading mathematicians. That ratio is the next decade of work: producing is cheap and abundant; trusting is slow, human and scarce.
Some Lean-checked; OpenAI warns unformalized ones “could have issues.” Verification abundance, adjudication scarcity.
Faros AI. LinearB (8.1M PRs): AI changes wait 4.6× longer, accepted 32.7% vs 84.4%.
Real progress. Someone still has to find the other 45% before the work can be used.
Machine output arrives without reasoning a reviewer can interrogate. It looks locally clean and gives no clue where it’s wrong.
A prover confirms the proof proves its statement; tests confirm what tests check. Neither confirms it’s what was needed.
Contracts are signed, designs stamped, papers defended. Responsibility is institutional — you can’t hold a model to it.
of AI-agent pull requests got no human review at all (EASE 2026). Zero-review merges up 31.3% (Faros).
of reviewers deliberately deprioritise AI changes (LinearB). Good machine work waits behind bad.
OpenAI chose which maths families were significant. When referees can’t keep up, the producer’s filter becomes the review.
Budget review hours next to model spend.
Provers, types, tests, policy engines.
Experts only where consequences are high.
Who profits from generation pays for checking.
Keep some production human for learners.
The first automation question was which jobs AI would do. The better one is which jobs AI makes more necessary: the ones that check, adjudicate and take responsibility. Expect a referee premium — senior engineers, auditors, specialist lawyers, reviewing scientists become the binding constraint on how much AI output anyone can use.Accountability — standing behind a result — may be the most durable form of human work there is.
Review Capacity May Limit AI Use
If AI output grows faster than review capacity, organisations may have to choose between slowing deployment, accepting more risk, or directing scarce expert time toward the most consequential work. The source describes three possible responses: unchecked output being allowed through, reviewers deprioritising AI-generated work, and producers deciding which of their own results merit attention. These are interpretations of the cited examples, not proof that every organisation is experiencing the same pattern.
The issue reaches beyond productivity metrics. In professional settings, contracts, engineering designs and scientific findings can carry legal, safety or reputational consequences. A model may assist with drafting or analysis, but people and institutions still have to decide whether the result is fit for use and who is responsible if it is not. That makes experienced reviewers a potential limit on the practical value of faster generation.
Products Worth Considering
BowFlex Results Series SelectTech Adjustable Dumbbells, Pair
Fitbit Sense Advanced Smartwatch – Sage Grey/Silver (One Size, S & L Bands Included)
Philips Sonicare DiamondClean 9900 Prestige Electric Toothbrush
As an affiliate, we earn on qualifying purchases.
Evidence Across Three Workflows
The examples cover mathematics, software and contract work, but the evidence comes from different kinds of sources and should not be treated as a single controlled comparison. The mathematics figures describe one OpenAI programme. The software statistics come from commercial analytics firms and a peer-reviewed study, while the contract example reports results against a set of evaluation criteria. Together, they illustrate the analysis’s central distinction: producing an answer and establishing that it is correct are separate tasks.
Some software sources cited, including Faros AI and LinearB, sell code-review products, the source acknowledges. Their commercial interests are a reason to examine the precise methods and figures carefully. The analysis says the findings point in a consistent direction, but the numbers alone do not establish that AI adoption caused every increase in review time or every difference in acceptance rates.
Verification tools can check parts of a result. A proof assistant can test whether a proof follows from a stated theorem, and software tests can check specified behaviour. But neither necessarily determines whether the theorem or tests captured the problem people meant to solve. The source argues that judgment about the question, relevance and consequences remains a distinct human task.
How Large Is the Review Gap?
The cited figures do not establish how widespread the review bottleneck is across industries, or whether the same effects occur in organisations with different workflows. The source gives no common measurement method linking the mathematics, software and contract examples. It also does not specify enough detail to assess the models’ performance across all tasks or the downstream consequences of their errors.
For software, the commercial interests of some data providers and the varied measures—review time, acceptance rates and absence of human review—make direct comparison difficult. The reported percentages should be read with their stated contexts, not as proof that AI-generated work is generally less reliable. It also remains unclear whether better tools, different review processes or more formal checks can reduce the pressure without shifting risk elsewhere.
The source raises a further concern about training: if junior staff do less drafting and coding themselves, they may have fewer opportunities to develop the judgment needed for senior review. That is a plausible concern, but the material does not quantify whether this is already reducing the supply of experienced reviewers.
Track Reviews and Training
The next useful evidence will be independent, comparable measurements of review time, error rates and outcomes across AI-assisted and human-produced work. For the cited mathematics programme, the distinction between formally checked results and unformalized manuscripts will remain relevant as results receive further scrutiny. In software and contract work, organisations will need to report how review is assigned and what happens when reviewers identify errors, not only how much output is produced.
For employers, the analysis points to a practical question: whether they can expand AI use while preserving routes for junior staff to learn the underlying work. The source does not prescribe a particular policy or establish that reviewer shortages are inevitable. It argues that monitoring who checks AI output, how much time checking takes, and how professionals gain experience will help show whether verification is becoming a genuine constraint.
Key Questions
What is the main claim of the analysis?
It argues that AI can lower the cost and time required to generate work while leaving the effort needed to check its correctness and suitability largely unchanged. The source presents this as a possible verification bottleneck, not as a measured condition in every field.
What does the source report about OpenAI’s mathematics work?
It says a model was given about 4,000 problems and produced 722 manuscripts in 372 families. Some results were formally checked in Lean; OpenAI cautioned that some unformalized results could have issues.
Do the software figures prove AI code is less reliable?
No. The figures cited concern measures such as review delays and acceptance rates, not a single direct measure of code correctness. The source also notes that some data providers sell review tools, so the findings and methods need careful scrutiny.
Why might human review remain necessary if AI can check AI?
Automated checks can test whether work meets specified rules, but may not establish whether the rules reflect the real task or whether the result is appropriate. The analysis also points to accountability: people and institutions, rather than models, bear professional and legal responsibility.
What remains uncertain?
The material does not establish how widespread the review gap is, whether AI caused the reported changes, or how much formal checking and improved workflows can reduce the burden. It also does not quantify whether AI use is changing the development of future expert reviewers.
Source: ThorstenMeyerAI.com
