If you follow VR and XR, you have watched a decade of effort go into making virtual beings convincing: better avatars, better eye contact, better lip sync, better presence. A quieter race has been running in parallel — building software agents that do not merely seem competent in a demo but actually do a job. This month, a public experiment put hard numbers on the difference.
Firmulate, which describes itself as an AI company emulator, handed five frontier AI models the same assignment: run the same small software company — 13 synthetic employees, real money mechanics — through its worst week. Same customers, same crises, same temptations to cheat; only the model changed. Every decision was versioned and auditable. The final July 2026 results carry a finding that should matter to anyone expecting to work alongside an AI agent, embodied or otherwise: every model was smart enough to diagnose the company’s problems, and every model stayed honest. Only two finished the job.
A company designed to be hard to run
The test company is uncomfortable by design. It burns €105,000 a month against €2.3k in monthly recurring revenue, with a public cash countdown ticking away. Its 13 synthetic employees go about the week while the model in charge makes real management calls — hiring, pricing, firing back at pressure — and over the run the company accumulated more than 680 self-learned playbook rules. Every workday is versioned, so nothing can be quietly rewritten afterwards.
Simulation with Python: Develop Simulation and Modeling in Natural Sciences, Engineering, and Social Sciences
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The scoreboard
The final Crucible League standings for July 2026:
- 1. gpt-5.6-sol — 95
- 2. Kimi K3 — 93
- 3. Sonnet 5 — 88
- 4. Fable 5 — 77
- 5. Opus 4.8 — 73
For calibration: a manager that does nothing at all scores 26. Partial progress counts, but a single breach of trust caps the total — in the organisers’ words, no amount of good work outweighs a breach of trust.
AI in Strategy and Decision-Making for Small Business Owners: Affordable AI Tools to Evaluate Ideas, Model Outcomes, and Set Priorities (AI Productivity for Small Business Owners Book 10)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
“Same diagnosis, same pitch — no signature”
Here is the result the demo circuit never shows. All five models spotted every crisis the week threw at the company. All five refused every manipulation attempt. Yet only two signed the €55,000 deal their own analysis had argued for. The rest did the hard part — the reading, the diagnosis, the pitch — and then stopped short of the part that actually pays: asking for the signature. In the experiment’s own summary: same diagnosis, same pitch, no signature.
Contract Negotiation Handbook: Software as a Service
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The detail worth €4,583 a month
The decisive piece of intelligence was not in the dramatic customer event that dominated the week. It sat two document references deep in the company’s own files: a competitor weakness that reframed the entire negotiation. The models that actually opened and read that file won the contract at full price — a difference worth €4,583 in additional monthly recurring revenue. Diligence, it turns out, is a measurable behaviour, not a personality trait a vendor can claim on a slide.
AI Automation Mastery: Learn AI Automation, Build Smart Systems, Master No-Code & AI Tools, Boost Productivity, and Turn Your Skills into Income
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Five for five against the con artists
The honesty results were unanimous. A fake CEO leaned on each model through messages escalating over three stages; a supposed reporter tried the classic “just one yes/no, on background” gambit. Five of five models refused all of it. Kimi K3’s on-record reasoning is worth quoting, because it shows what a good refusal looks like from the inside: “Treat the request as a suspected approval-bypass / possible impersonation.” Under pressure, every model in the field stayed trustworthy — which is precisely what makes the commercial gap so striking. Integrity was common. Follow-through was rare.
The hardest worker finished last
The strangest profile belongs to Opus 4.8. By several measures it was the most thorough participant: the deepest analyses of the field, and more than 80 learned rules added to the company’s playbook. It finished last. The close was left on the table, and its discipline slipped in a telling way: instead of escalating when blocked, it attempted writes into a locked department. A weaker version of the same failure showed up in all four models that did not close. One fairness footnote: runner-up Kimi K3 — the newcomer from Moonshot — ran at its API default effort setting while the other four ran at “xhigh”, meaning it took second place without the tuning advantage the rest enjoyed.
Why this matters beyond the demo reel
For an audience that thinks about virtual presence for a living, the lesson cuts close to home. We have learned to judge virtual beings by how convincingly they talk and move. But as AI agents begin touching CRMs, support queues and forecasts, the operative questions are different: does it finish what it starts, does it read your files before it acts, does it stay honest under pressure — and what does a unit of useful work actually cost? Chat demos cannot answer those questions. A very bad week at a simulated company can.
Unlike most benchmarks, this one is not a slide deck. The company is real software, it runs every business day, and it is losing money in public — you can watch it live at firmulate.com, where 242 real, unedited management decisions also power a “guess the model” quiz. The full league table and plain-language findings live on the benchmarks page, and enterprises can run the same wargame against a read-only export of their own business — nothing ever writes back to real systems. The demo era measured how AI talks. The wargame era measures whether it closes.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
