Most AI benchmarks are basically standardized tests. And just like standardized tests, they’re great at measuring how well you take standardized tests. Not so great at measuring whether you can actually do the job. A new research collaboration called CRUX is trying to fix that by testing AI agents on long, messy, real-world tasks. Their first experiment? They had an AI agent build and publish an iOS app to the App Store. It worked.
If you run a service business and you’re making bets on AI, this matters. A lot. Because the gap between “scores well on benchmarks” and “actually useful in production” is exactly where your money gets wasted.
What happened
- A team of 17 researchers from academia, government, civil society, and industry launched CRUX (Collaborative Research for Updating AI eXpectations), a project for regularly running open-world evaluations of frontier AI.
- Open-world evaluations test AI agents on real, end-to-end tasks instead of narrow benchmark puzzles. Think: building a C compiler that can compile the Linux kernel, or running a small shop, or publishing an app.
- In CRUX’s first experiment, an AI agent developed and published a simple iOS app to the App Store. It made just two errors, one of which required a human to step in (it forgot where credentials were stored and made up a fake phone number for Apple’s review process).
- The researchers disclosed results to Apple a month before publishing, warning that autonomous app store spam could become a real problem soon.
- The team plans to run more open-world evaluations across AI R&D automation, AI governance, and other domains.
The numbers
- The iOS app experiment cost roughly $1,000 total, though the actual app development and submission only cost $25. The rest was spent on tokens for monitoring the app’s status.
- Anthropic’s earlier open-world eval (having Claude build a C compiler) cost around $20,000.
- The agent made 2 errors during the iOS app process. Only 1 required manual intervention.
- Many prominent AI benchmarks have seen successor versions released in just the last two years because the originals got saturated so quickly.
5 things service business operators should know about open-world evaluations
- Benchmarks lie in both directions. They can overestimate AI capabilities (models get trained to ace the test) and underestimate them (a CAPTCHA can tank a score even when the agent can do the underlying work). Neither extreme helps you make good decisions about where to deploy AI.
- Reliability matters more than capability. The CRUX researchers point out that even though agents are improving fast on capability metrics like accuracy, they’ve improved much more slowly on reliability. Translation: your AI can do impressive things sometimes, but doing them consistently is a different story.
- The cost of real-world AI tasks is dropping fast. The iOS app experiment cost $25 in actual development tokens. A year ago, comparable experiments cost 100x more. That curve isn’t flattening.
- “Sample size of 1” still has value. Open-world evals often test a single complex task rather than hundreds of simple ones. That feels unscientific. But one successful end-to-end run of a real task tells you something a thousand benchmark scores can’t: that the capability exists in the wild, and it’s coming for your industry.
- Spam and abuse risks are real. If an AI can autonomously publish an app for $25, it can publish a thousand of them for $25,000. The researchers flagged app store spam as an immediate concern. Every platform that accepts submissions (legal filings, insurance claims, permit applications) should be thinking about this.
The hot take
Benchmarks have become a vanity metric for AI labs. They’re useful for comparing models against each other, but they’re nearly useless for predicting what an AI agent will actually do when you point it at a real business process. Open-world evaluations aren’t a cute academic exercise. They’re the only honest way to figure out what these systems can and can’t do today. If you’re choosing AI tools based on leaderboard scores, you’re basically hiring someone because they aced the SAT. That’s not how this works.
The Agency OS play
Before you spend another dollar on an AI agent or copilot, run your own version of an open-world evaluation. Pick one real, messy, end-to-end task from your business. Not a demo. Not a toy problem. Something with logins, forms, approvals, and weird edge cases. Hand it to the AI tool you’re considering and watch what happens. Log everything. Note where it fails, where it needs a human, and where it surprises you.
If you’re building agent-powered products or workflows for clients, steal the CRUX playbook. Specify exactly what human intervention is allowed before the test starts. Release (or at least review) the full logs of what the agent did. Analyze failures qualitatively, not just pass/fail. This is how you avoid shipping something that looks great in a demo and falls apart in production. Your clients will trust you more for it.
Finally, if your business runs any kind of intake process that accepts outside submissions (applications, claims, proposals, listings), start planning for autonomous spam now. The $25 app submission is a preview. Build verification steps, rate limits, and human review checkpoints before you’re drowning in AI-generated junk. The warning signs are here. The wave is next.