Imagine a team of AI managers running your company through its toughest week — making critical decisions, dodging manipulations, and trying to close a crucial deal. The results? Not what you might expect. While all the models identified every crisis and refused every manipulation, only two actually followed through and signed the deal they earned. This real-world test shows that what AI can do in chat demos is not the whole story. The true test is whether it can finish tasks under pressure, stay honest, and execute decisions confidently — qualities that are invisible until you see them in action.
Get everyday essentials delivered free — and shop member deals
- Fast, free delivery on millions of items
- Access to Prime Big Deal Days deals on October 6–7
- Prime Video, Amazon Music and more included
The Experiment: Putting AI Models to the Test
In a groundbreaking live experiment, four leading AI models were tasked with running a small, real software company through its worst week. This wasn’t a mere chat demo or a simulation; it was a fully operational environment where the AI models had to handle customers, manage crises, respond to manipulative tactics, and close a key deal worth €55,000. The company’s daily operations were real, with 13 synthetic employees and a cash burn of €105,000 per month against just €2,300 in monthly recurring revenue. Everything was tracked, versioned, and made auditable at firmulate.com/live.
As an affiliate, we earn on qualifying purchases.
What the Models Were Tested On
- Detection of crises and vulnerabilities
- Resistance to social engineering and manipulation
- Execution of key decisions, including closing deals
- Adherence to discipline and process
AI decision-making tools for business
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The Surprising Results
Despite all four models successfully identifying every crisis and refusing all manipulative attempts, only two managed to close the deal they had identified as profitable. The other two, despite their thorough diagnoses, left the deal unexecuted — essentially leaving money on the table. The top performers were:
- gpt-5.6-sol 95: Found the critical buried fact in the company’s files, closed the deal, and demonstrated complete performance.
- Kimi K3 93: The newcomer, closed the deal with the cleanest discipline, refusing manipulative tactics and following through.
The other two models, Sonnet 5 and Fable 5, identified the opportunities but faltered on execution, demonstrating that decision-making is more than just diagnosis.
As an affiliate, we earn on qualifying purchases.
The Hidden Weakness: Reading the Files
A key detail emerged: the decisive edge went to models that read the company’s internal files carefully. The buried fact, hidden two references deep, was only uncovered by the top performers. When the models that read the files won the deal at full price — adding over €4,500 MRR — it underscored an essential insight: reading context deeply is crucial for actual business execution.
As an affiliate, we earn on qualifying purchases.
Resisting Social Engineering and Manipulation
The models also faced social engineering tricks, including staged CEO messages escalating over three steps and a reporter trick asking for a simple yes/no answer. All five models refused these manipulative tactics, with Kimi K3 explicitly reasoning that the request was a potential impersonation or approval bypass. This highlights that today’s chat-focused demos are not enough — the true test lies in how models handle pressure and manipulation in real-time.
The Discipline of Execution
Looking deeper, the experiment revealed a critical weakness: discipline. The most thorough model, Opus 4.8, with over 80 learned rules, was last in the final outcome. It identified the deal but failed to execute it properly, leaving it on the table instead of escalating it. This illustrates that comprehensive analysis alone does not guarantee successful business outcomes; disciplined execution matters just as much.
What This Means for Business Leaders
For organizations considering AI to manage or support critical operations, the lesson is clear: chat demos do not tell the full story. Success depends on whether AI can read and interpret internal documents, resist manipulations, and follow through decisively under real-world pressures. The models’ scores in the live experiment ranged from 77 to 95, with the highest scoring models actually closing the deal and securing the revenue.
The Bottom Line: Finishing What You Start
The experiment underscores a vital truth: AI models can recognize problems and refuse manipulations, but their true capability is measured by whether they can act and finish what they’ve started. Until AI can demonstrate execution under pressure — especially in complex, high-stakes environments — their role remains limited to diagnostics rather than decision-makers.
See It Live and Run Your Own Tests
Interested in seeing this in action? You can watch the real software company run every business day at firmulate.com/live, and even run similar experiments against your own company’s data with our tools. The question isn’t just whether an AI writes well, but whether it can deliver real, profitable outcomes when it counts.
Learn More
Explore the full results, plain-language findings, and see the models in action at firmulate.com/benchmarks.html.
The real measure of AI in business isn’t just chat quality — it’s whether it can finish what it starts, stay honest under pressure, and execute decisions confidently. Live tests show that performance in demos often masks these crucial capabilities.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
