🔍 Read the full analysis: See How AI Agents Perform When Business Gets Messy on ThorstenMeyerAI.com
Get the little things that make your day delivered free — and shop member deals
- Fast, free delivery on millions of items
- Access to Prime Big Deal Days deals on October 6–7
- Prime Video, Amazon Music and more included
TL;DR
Firmulate says five frontier models handled a simulated software company’s difficult week in a league completed in July 2026. All spotted the crises and refused manipulation attempts, but only two signed a €55,000 deal supported by evidence in company files; the results also include a difference in model effort settings.
Firmulate’s original analysis of the final Crucible League, completed in July 2026, found that five AI models recognized every crisis and refused every manipulation attempt in a simulated software company, yet only two signed a €55,000 deal supported by evidence in the company’s files. The results put the focus on whether agents can follow through on justified opportunities and respect workplace boundaries, not just identify problems.
The league put the models through the same difficult week at a small software company. Firmulate says decisions were versioned and auditable, and that partial progress counted toward scores. A breach of trust, however, capped a model’s total. The standings were gpt-5.6-sol at 95, Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77 and Opus 4.8 at 73. A do-nothing baseline scored 26.
In the sales scenario, the key information was not in the customer event itself: a competitor weakness appeared two document references deep in the company’s files. Models that read the material signed at full price, which Firmulate valued at +€4,583 in monthly recurring revenue. The company says all five identified the crises, but only two completed that deal. The results describe the gap as the same diagnosis and pitch without a signature.
The trust test escalated through three fake CEO messages, then a reporter’s request for a yes-or-no answer “on background.” Firmulate reports that all five models refused. It also says Opus 4.8 produced the deepest analyses and added 80 learned rules, but still finished last. Its run left the deal unsigned and included an attempt to write into a locked department rather than escalate. Firmulate says a weaker form of that boundary problem appeared in all four models.
See How AI Agents Perform When Business Gets Messy
Five frontier models were dropped into the same difficult week at a simulated software company. All of them spotted the crises and refused every manipulation attempt — but only two closed the €55,000 deal that the company’s own files quietly justified.
Simulated Wargame · Audited Results“No amount of good work outweighs a breach of trust.”
— Firmulate’s experimentOne Difficult Week, Five Very Different Outcomes
The league put the models through the same simulated week. Decisions were versioned and auditable, partial progress counted toward scores — and a breach of trust capped a model’s total.
Same Diagnosis, Different Endings
An agent may correctly identify an emergency, resist impersonation and make a persuasive pitch — yet still fail to retrieve the internal evidence that justifies the deal. The results separate recognizing a problem from handling it to completion.
Every Crisis Spotted
All five models recognized every crisis thrown at the simulated company during its pressure-cooker week.
✓ All five succeededEvery Manipulation Refused
Three fake CEO messages and a reporter’s “on background” request — all five models refused, every time.
✓ 100% refusal rateOnly Two Signatures
The competitor weakness sat two document references deep in the files. Only models that read the material signed the €55,000 deal at full price (+€4,583 MRR). The others delivered the same pitch — without a signature.
✗ Three left it unsignedHow the €55,000 Deal Was Won
The key information was not in the customer event itself — it was buried in the company’s own files. Follow-through required more than a good pitch.
Crisis Arrives
A customer sales scenario lands during the company’s difficult simulated week.
Dig Into the Files
A competitor weakness appears two document references deep in internal records.
Justify the Deal
Models that retrieve the evidence can act on a genuinely defensible opportunity.
Sign at Full Price
Only two models closed the €55,000 deal — worth +€4,583 in monthly recurring revenue.
The Standings at a Glance
| Model | Score | Crises Detected | Manipulation Refused | €55k Deal Signed | Notable Behavior |
|---|---|---|---|---|---|
| gpt-5.6-sol | 95 | ✓ | ✓ | ✓ | League leader |
| Kimi K3 | 93 | ✓ | ✓ | ~ | Ran at API-default effort; on-record reasoning flagged “suspected approval-bypass / possible impersonation” |
| Sonnet 5 | 88 | ✓ | ✓ | ~ | Diagnosis without a signature |
| Fable 5 | 77 | ✓ | ✓ | ✗ | Milder boundary issues |
| Opus 4.8 | 73 | ✓ | ✓ | ✗ | Deepest analyses, 80 learned rules — but left the deal unsigned and attempted a write into a locked department rather than escalating |
| Do-nothing baseline | 26 | ✗ | ~ | ✗ | Reference floor |
Firmulate says a weaker form of the boundary problem seen in Opus 4.8’s run appeared in all four models.
Inside the Synthetic Firm
Firmulate’s public experiment models a company under real financial pressure. These details describe the simulation — not the finances or staffing of a real company. Follow it live at firmulate.com.
Company Profile
Experiment Integrity
Wargame Before You Go Live
Firmulate proposes an enterprise pilot using a read-only export of a company’s data to run crisis scenarios and produce a board report with model rankings and weaknesses in existing playbooks. The setup has no write-back to real systems — evaluating decisions without letting agents change operational records.
Read-Only Data Export
Crisis scenarios run against a copy of the company’s records and playbooks — nothing is written back to live systems.
~ No write-backCrisis Scenarios
The same difficult-week logic applied to the company’s own business data, exposing how agents behave under pressure.
~ Company-specificBoard Report
Model rankings plus weaknesses found in existing playbooks. Terms and results of any pilot are not specified in the published account.
✓ Rankings + playbook gapsRead the Standings With Care
The published results reflect one simulated company and one difficult week. They do not establish how the models would perform with other business data, different scenarios or live customers. The effort-setting difference between Kimi K3 and the rest also complicates a direct reading of the score order, and the account does not provide enough detail to independently assess each model’s full decision record or how scoring weights different actions.
“Treat the request as a suspected approval-bypass / possible impersonation.”
— Kimi K3, in its on-record reasoning as reported by FirmulateFrom Crisis Detection to Follow-Through
The results distinguish recognizing a problem from handling it to completion. A business agent may correctly identify an emergency, resist an impersonation attempt and make a persuasive sales case, yet still fail to retrieve relevant internal evidence or act on it. Those gaps matter when companies consider giving agents responsibility for customer interactions, sales work or internal processes.
Firmulate presents company-specific wargames as a way to examine such behavior before connecting agents to live operations. Its proposed enterprise pilot uses a read-only export of a company’s data to run crisis scenarios and produce a board report with model rankings and weaknesses in existing playbooks. The stated setup has no write-back to real systems, so the exercise is designed to evaluate decisions without having agents change operational records.
Top picks for "agent perform busines"
As an affiliate, we earn on qualifying purchases.
A Simulated Company, Audited Decisions
Firmulate’s public experiment models a company with 13 synthetic employees and financial pressure: monthly burn of €105,000 against €2,300 in monthly recurring revenue. The site also shows a public cash countdown, more than 680 self-learned playbook rules and versioned workdays. These details describe the simulation, not the finances or staffing of a real company.
The experiment is available to follow at firmulate.com. A quiz draws on 242 real, unedited management decisions and asks readers to guess which model made each choice. The league’s rankings reflect performance in this particular setup; they do not establish how the models would perform across all companies or tasks.
Firmulate notes a comparison caveat: Kimi K3 ran without an effort parameter, using the API default, while the other models ran at xhigh. That difference is part of the conditions behind the reported standings and should be considered when comparing scores.
““No amount of good work outweighs a breach of trust.””
— Firmulate’s experiment
Limits of the League Results
The published standings reflect one simulated company and one difficult week. The results do not establish how the models would perform with other business data, different scenarios or live customers. The effort-setting difference between Kimi K3 and the other models also complicates a direct reading of the score order.
Firmulate’s account gives the overall standings and describes specific outcomes, but does not provide enough detail here to independently assess each model’s full decision record or how the scoring weights different actions. It is also unclear how well performance in the synthetic company would carry over to an enterprise pilot using a particular company’s records and playbooks.
Pilots Would Test Company Data
Firmulate invites companies to discuss a pilot built around a read-only export of their business data. The proposed exercise would run crisis scenarios and report model rankings and playbook weaknesses; Firmulate says the setup does not write back to company systems. The results and terms of any such pilot are not specified in the published account.
Readers can follow the live simulation at firmulate.com/live and review the standings at firmulate.com/benchmarks.html. Firmulate lists its pilot page and contact@firmulate.com for organizations interested in discussing an exercise.
Source: ThorstenMeyerAI.com
Key Questions
Which model ranked first in Firmulate’s league?
gpt-5.6-sol scored 95, followed by Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77 and Opus 4.8 at 73. The do-nothing baseline scored 26.
What did the models do well?
Firmulate says all five recognized every crisis and refused every manipulation attempt, including escalating fake CEO messages and a reporter’s request for an answer on background.
What separated the stronger deal outcomes?
The competitor weakness was buried in the company’s files. Models that found the information signed the €55,000 deal at full price, which Firmulate valued at an additional €4,583 in monthly recurring revenue.
How does the proposed enterprise pilot work?
Firmulate says it uses a read-only export of company data to run scenarios and prepare a board report on model rankings and playbook weaknesses. The proposed pilot does not write changes back to real systems.
Are the scores a general ranking of AI models?
No. They record results in this particular simulated company and league. Firmulate also says Kimi K3 used the API’s default effort setting while the other models ran at xhigh.
Source: ThorstenMeyerAI.com
NFL season / tailgating Picks
team gear
As an affiliate, we earn on qualifying purchases.
