What's fundamentally different about AutoLab compared to the familiar SWE-bench or LiveCodeBench?
The core difference lies in the task's time span and the nature of the interaction. Static benchmarks like LiveCodeBench are essentially one-shot: give the model a question, see if it can produce the correct answer in one go, with no back-and-forth at all. SWE-bench-style benchmarks push this a step further, allowing multi-turn interaction, reading error messages, and revising code — but the entire task typically wraps up within minutes to half an hour, closer to "fix a well-defined bug."
AutoLab tests a categorically different capability. The model isn't handed a clear error that needs "fixing" — it's handed a baseline implementation that runs but is deliberately poor, with the goal of sustained optimization over a wall-clock budget stretching from hours to over a dozen hours. That means the model has to decide for itself which direction to explore next, when to abandon a given attempt, and how to allocate its time budget — decisions that almost never come up in short-horizon evaluations, yet form the core of what real-world research and engineering work actually looks like.
Why is the "agent harness" variable worth calling out specifically?
Because it exposes a blind spot that's frequently overlooked in benchmarking: a model's score doesn't purely reflect the model's own capability — it also reflects how well-designed the operational framework is that the model uses to interact with its environment. The AutoLab team's experiment showed that re-testing the same batch of models under a harness originally designed for one-shot patch editing caused models to submit prematurely and abandon further optimization, producing a noticeably lower score — a drop that has nothing to do with the model's actual capability and is purely an artifact of a mismatch between the harness and the task's nature.
The significance of this finding is that if a long-horizon benchmark's leaderboard doesn't disclose which harness was used and whether that harness was properly adapted to the task, seeing a model score low doesn't automatically mean that model is "bad at long-horizon tasks" — it might just be the harness dragging it down. That's exactly why the AutoLab team specifically argues that future long-horizon evaluations should treat harness selection and design as a variable worth disclosing on equal footing with the model's own capability.
How does the "Dominance" metric differ from a simple average score, and why did the team design this metric separately?
The average score (Avg@3) reflects a model's absolute performance level across all tasks, but that number can be dragged down or inflated by a handful of unusually extreme tasks — a model that happened to score zero on one particular problem, for instance — and doesn't directly show how often a model actually wins when going head-to-head against other specific competitors.
Dominance measures things from a different angle: for every task, it directly compares whether a given model's score beat each other model's score — a win scores 1, a tie scores 0.5 — then averages that across every task and every opponent. The advantage of this metric is that it's relatively insensitive to hardware performance differences and per-task scoring design quirks, offering a tournament-style perspective that's less likely to be dominated by a small number of high-Leverage tasks with unusually large score swings. claude-opus-4.6's overall Dominance score of 0.93 means it held an advantage in nearly every task against nearly every opponent — a more intuitive way to convey how far ahead it was than the 0.68 average score alone.
Ordinary readers won't be running this kind of benchmark themselves — what practical use does this research have for everyday judgments about AI products?
The most direct benefit is correcting a common misconception: seeing a model score high on a coding benchmark leaderboard doesn't mean that model is well-suited to tasks requiring sustained autonomous work — running an AI Agent on a research project for hours, or automatically optimizing a system, for example. This study shows that "getting it right in one shot" and "iterating persistently until it's actually good" are two distinct capabilities that need to be evaluated separately, and the scores most product marketing cites usually only cover the former.
In practice, if you're evaluating an AI product or service marketed as able to "autonomously complete long-horizon tasks," more useful questions to ask include: has this company publicly disclosed how often its model abandons or prematurely ends long tasks midway through, has it disclosed the details of the harness used in its evaluations, and has the product actually been specifically optimized for effective time-budget allocation — rather than simply repackaging a model that's good at one-shot answers as something capable of working autonomously for hours.
In June 2026, a cross-institutional research team released a new benchmark called AutoLab, specifically designed to test whether frontier models can work like real researchers — spending hours, sometimes over a dozen hours, cycling through "inspect the code, propose a change, run the experiment, read the result, refine again" — rather than being judged, as most existing benchmarks do, on whether they can get a single question right in one shot. The benchmark's conclusion cuts against most people's intuition about which model is "smartest": what actually separates models isn't the elegance of anyone's first attempt — it's whether a model is still willing to make a 47th revision after being rejected 46 times.
The research team notes that existing frontier model evaluations broadly fall into two categories: static, single-turn benchmarks like LiveCodeBench, which test whether a model knows how to write correct code but involve only a single round of interaction; and short-horizon interactive benchmarks like SWE-bench, where a model can go back and forth, but the entire task typically wraps up within minutes to half an hour. Genuinely long-horizon, closed-loop optimization tasks that unfold over hours and require a model to judge for itself whether to keep digging deeper or pivot to a different approach had, until now, only been partially touched by a handful of scattered evaluations, most confined to a single domain (GPU kernel optimization only, or ML engineering only) — none covering multiple real research and engineering domains simultaneously, while also being difficult enough that even today's strongest models can't easily max out the score.
AutoLab consists of 36 tasks personally contributed by senior researchers and engineers, spanning system optimization (15 tasks covering kernels, sorting, hashing, compression, cryptography, and other low-level performance engineering in C, Rust, Go, and Python), model development (7 tasks spanning the full LLM development pipeline, including pretraining scaling laws, RL post-training, and SFT data selection), puzzle & challenge (10 algorithmic problems each built around a single key insight), and CUDA (4 tasks targeting GPU kernel optimization for cryptographic primitives, point-cloud registration, and compression). Every task ships with a baseline that's correct but deliberately suboptimal, and the model has to keep improving that code within a strict wall-clock budget — ranging from 2 hours for the smallest puzzle tasks to 12 hours for end-to-end LLM development tasks.
The team built in a set of anti-shortcut mechanisms: the verifier used for scoring is entirely sealed off from the model, which only gets a local evaluation script for its own development-stage self-checks; ML tasks carry an additional correctness gate whose input data is drawn from a distribution completely disjoint from anything visible during development; the team also ran a dedicated adversarial agent explicitly tasked with hunting for any exploitable shortcut during task design; and critical files that shouldn't be touched are SHA-pinned, with any unauthorized modification triggering an automatic zero. The full evaluation run consumed 2,544 wall-clock hours and 8.6 billion tokens.
On the overall composite score (Avg@3, the average across three independent trials), claude-opus-4.6 topped all four sub-domains with a score of 0.68, while the runner-up (which rotates between gemini-3.1-pro and kimi-k2.6 depending on the domain) landed around 0.50 — a meaningfully wide gap. Even more notable is the "Dominance" metric, which measures a model's head-to-head win rate against every other model across every task: claude-opus-4.6 scored 0.93 on this metric, meaning it held an advantage over the vast majority of tasks against the vast majority of rivals. By contrast, most other frontier models' Dominance scores fell between 0.3 and 0.6, including gpt-5.4 — previously seen as a strong contender — which scored just 0.39. The research team specifically noted that several otherwise capable models, including gpt-5.4, underperformed on this benchmark for reasons unrelated to raw coding ability: some terminated exploration prematurely after only a few steps, while others exhausted their entire time budget without producing a valid final solution at all.
The research team conducted an in-depth analysis of every trajectory, including a manual review of 302 zero-score rollouts, arriving at the study's central finding: final performance correlates far more strongly with a model's persistence — repeatedly benchmarking, editing code, and incorporating empirical feedback throughout the trajectory — than with the quality of its first-attempt solution. In other words, a model that starts with a mediocre implementation but keeps running tests and revising based on results will typically end up far ahead of a model with an impressive opening move that never adjusts course based on empirical feedback. The team summarizes this as: long-horizon optimization capability requires time awareness and sustained empirical iteration — a distinct capability that can't simply be inferred from how "smart" a model is, and needs to be measured separately.
The AutoLab team also ran an experiment most benchmarks skip entirely: re-testing the same batch of models under different "agent harnesses" — the operational framework a model uses to interact with the code environment — comparing the Harbor framework paired with the terminus-2 agent (the team's default combination) against two alternative harnesses, mini-swe-agent and pi-mono. The results showed that even for the exact same model, switching harnesses could produce a meaningful swing in score — one harness, originally designed for one-shot patch-style editing, proved poorly suited to this kind of sustained-iteration optimization task and pushed models toward submitting prematurely and abandoning further optimization. This means benchmark scores aren't just shaped by a model's own capability, but also by the design of the harness paired with it during evaluation — a variable that's frequently dismissed as "a mere implementation detail" and simply overlooked.
For readers evaluating AI-related investments or choosing AI tools, this study's most direct takeaway is: if what you actually care about is whether a model can handle tasks that genuinely require sustained autonomous work — automated research, systems optimization, complex engineering projects — a model's score on a traditional one-shot coding benchmark like SWE-bench has fairly limited reference value, because it's measuring a fundamentally different capability dimension. A more useful signal to watch for is whether a company publicly discloses how its models behave on long-horizon tasks (does it tend to give up early, does it allocate its time budget effectively), and whether the harness design used in its evaluations is transparent and reproducible by third parties — since the harness itself can meaningfully move the score, a long-horizon benchmark result whose harness details were never disclosed deserves to be trusted a little less.