Lead · Milestones

What Separates Success From Failure Isn't How Clever the First Attempt Is — It's Whether the Agent Tries a 47th Time: What a 2,544-Hour Benchmark Revealed

What separates the top model from the rest isn't whose first attempt is smarter — it's who's still willing to try a 47th revision after 46 rejections.

Elena Voss · September 05, 2026

In June 2026, a cross-institutional research team released a new benchmark called AutoLab, specifically designed to test whether frontier models can work like real researchers — spending hours, sometimes over a dozen hours, cycling through "inspect the code, propose a change, run the experiment, read the result, refine again" — rather than being judged, as most existing benchmarks do, on...

Milestones

The Company That Built This Benchmark Just Declared It Broken: Why SWE-bench Can No Longer Tell You If AI Can Actually Code

Same models, a contamination-resistant leaderboard, and scores drop from 93%...

Milestones

How Long Can AI Work Autonomously? METR's Time Horizon Doubles in Months — But the Number Is Messier Than It Looks

Same model, same test — but depending on whether cheating counts as success,...