🔍 Read the full analysis: FrontierCode: Is Your AI’s Pull Request Ready For Maintainer Review? on ThorstenMeyerAI.com
Get the little things that make your day delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Artificial Analysis added FrontierCode, a benchmark that scores whether AI-generated code changes meet maintainer review criteria, and tested 23 models. Claude Opus 5.5 led with a 52.2% overall score and passed every blocking check in 57.7% of attempts, leaving substantial room for human review.
Artificial Analysis tested 23 AI models on FrontierCode, a coding benchmark that evaluates whether a patch would meet repository maintainers’ review standards. Claude Opus 5.5 ranked first with a 52.2% score, but it passed every blocking check in only 57.7% of attempts, suggesting that even the leading result does not establish reliable, hands-off pull requests.
The benchmark results, added to Artificial Analysis’ evaluations on October 12, 2026, rank Claude Opus 5.5 (max with fallback) first, followed by GPT-6.1 Sol at 49.1% and Claude Fable 5.1 at 49.0%. Artificial Analysis reports that Opus passed all blocking checks in 57.7% of attempts. The remaining attempts failed at least one blocking check; the source describes that failure rate as about 42%.
FrontierCode’s score is not simply a count of passing tests. The benchmark checks hidden-test correctness, regressions and cleanliness, whether an agent’s tests distinguish the changed code from the original, adaptive behavior tests, limits on files and diff size, and an LLM review against maintainers’ criteria. Failing any blocking check earns zero, while passing them all produces a score from 70% to 100%, depending on quality checks. Attempts flagged by Cognition’s trajectory scanner for looking up solutions also score zero.
Artificial Analysis ran each model through the same open-source agent harness, Stirrup, with a limit of 250 turns per attempt and three attempts per task. Its run used 100 tasks from FrontierCode’s Main set. Nine of those tasks were not solved by any tested model on any attempt, according to the source. GPT-6 Luna (medium) served as the judge for the review step.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Passing Tests Is Not Enough
The results address a gap between code that runs and code that a project maintainer is prepared to merge. A patch can pass ordinary tests yet still be poorly scoped, introduce regressions, include weak tests, or fail project-specific review criteria. FrontierCode is designed to account for those issues, making its scores a different signal from conventional benchmarks focused mainly on functional correctness.
For teams considering AI agents for repository work, the 57.7% blocking-check pass rate for the top model indicates that human review remains relevant in this test setting. The score is not a direct estimate of how often a model’s pull requests will be accepted in every real project: the benchmark uses a particular task set, harness and judging process. It does, however, show why a green test suite alone may not answer whether an AI-generated change is ready for review.
The results also show a cost trade-off. Artificial Analysis lists Claude Opus 5.5 at $10.15 per task and GPT-6.1 Sol at $0.81. Opus scores 3.1 percentage points higher, while costing about 12.5 times as much per task in this evaluation. Those figures may help teams compare options, but they do not account for every deployment cost or the value of different kinds of mistakes.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Grades Patches
FrontierCode was announced by Cognition in June 2026 and developed with open-source maintainers. Its version 1.1 followed in July with rules intended to prevent agents from finding task solutions online. The full benchmark contains 150 private tasks; Artificial Analysis evaluated models on the 100-task Main set. Keeping tasks private is intended to reduce the chance that models have encountered their solutions during training.
More than 20 maintainers contributed tasks drawn from 36 established repositories, including Celery, Budibase, uppy and Mattermost. The source says maintainers spent more than 40 hours on each task. The benchmark’s use of scope limits and review criteria is intended to penalize behaviors that can make a patch harder to evaluate, not just code that fails to work.
In Cognition’s announcement, Celery maintainer Tomer Nosrati described the approach this way: “Where others grade like a CI, FrontierCode grades like a tech lead.” The comparison captures the benchmark’s stated distinction: it combines automated checks with an assessment of whether a change meets maintainer-defined expectations.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, a Celery maintainer, in Cognition’s announcement
programmer testing and debugging software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
What the Scores Cannot Show
The reported results do not establish how often these models’ patches would be accepted across all repositories or in routine production use. The evaluation covers 100 tasks from a 150-task benchmark, uses a shared agent harness and includes an LLM review judge. How scores might change with different repositories, workflows or human reviewers is not established in the source.
Several models had no reported FrontierCode result as of October 13, including Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview. Their absence means the published ranking is not a comparison of every prominent coding model. The source also does not provide detailed results for each task or explain how closely the benchmark’s review criteria match different projects’ policies.
Cost figures are reported per task, but the source does not specify that they represent every expense involved in operating an agent. The results should be read as measurements within Artificial Analysis’ stated setup, not as guarantees of performance or total cost in other environments.
software development testing tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
More Models and Runs to Watch
The next useful development will be results for models that were missing from the October 13 leaderboard, particularly those named by the source as having no score. Future additions could change the ranking and provide a broader comparison, but no release dates or planned evaluation schedule are provided in the source.
Readers comparing models should watch for the benchmark version, task set, agent harness and cost assumptions attached to any new results. Until those details are available for additional models or repeated evaluations, the current findings remain a snapshot: Opus leads this run, yet its blocking-check pass rate also shows that review-ready performance is not assured.
As an affiliate, we earn on qualifying purchases.
Key Questions
What is FrontierCode measuring?
It measures whether an AI-generated code change meets a range of correctness, regression, test-quality, scope and maintainer-review checks—not only whether tests pass.
Which model ranked first?
Claude Opus 5.5 (max with fallback) ranked first in Artificial Analysis’ run with a 52.2% FrontierCode score. It passed every blocking check in 57.7% of attempts.
Does the result mean the top model can reliably submit merge-ready pull requests?
No. The reported pass rate shows that the top model failed at least one blocking check in a substantial share of attempts in this evaluation. The results do not establish performance across all repositories or real-world workflows.
How many tasks and models were included?
Artificial Analysis tested 23 models on the 100-task Main set. FrontierCode’s full benchmark has 150 tasks.
What remains missing from the comparison?
The source says Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of October 13, 2026. It also does not provide a date for when those results may be added.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
