On July 9, OpenAI shipped the GPT-5.6 family — Luna, Terra, Sol — claiming to beat Claude Fable 5 “by 13.1 points”. On July 24, Anthropic answered with Claude Opus 5 and its 96% on SWE-bench Verified. Two weeks, two press releases, an avalanche of percentages… and one legitimate question: what are we supposed to do with these numbers? That’s where you need to understand what a benchmark is — and above all, how to read one. You’ll see: it’s not rocket science.
The crash test for models
A benchmark is the Euro NCAP crash test of LLMs: a standardized trial, identical for every candidate, producing a comparable score. Like for cars, it rests on three ingredients:
- A frozen set of tasks: hundreds or thousands of problems, always the same — the concrete wall is in the same place for everyone.
- An automatic metric: a binary or numeric criterion that requires no human opinion — the crash dummy has sensors, not feelings.
- An execution harness: the protocol that presents the task to the model and collects its answer — the launch rail of the crash test.
The final score is a success percentage. “Fable 5: 95% on SWE-bench Verified” means: out of the 500 tasks in the set, the model solved 95%.
How it works in practice
Take SWE-bench, the flagship coding benchmark. Its tasks are real GitHub issues pulled from open-source projects: “fix this bug in Django”, with the full repository available. The flow:
- The model receives the issue and the repo’s code.
- It produces a patch.
- The harness applies the patch and runs the project’s own test suite — the tests that failed before and must pass after.
- Green tests = task solved. The score is the percentage of issues resolved.
The beauty of the system: grading is objective and automatable. No judge, no subjectivity — tests pass or they don’t. The same principle drives agentic benchmarks (OSWorld has the model drive a real Linux desktop, Terminal-Bench a real terminal): you measure a verifiable outcome, not an impression.
One detail that matters: most published scores are pass@1 — a single attempt per task. A “pass@8” (eight tries, keep the best) mechanically inflates the number; always check the fine print under the table.
The numbers of the moment
Two freshly released families, and what the announcements highlight:
| Model | Price (in/out per MTok) | SWE-bench Verified | SWE-bench Pro |
|---|---|---|---|
| Claude Fable 5 | $10 / $50 | 95.0% | 80.3% |
| Claude Opus 5 | $5 / $25 | 96.0% | 79.2% |
| GPT-5.6 Sol | $5 / $30 | — | 64.6% |
| GPT-5.6 Terra | $2.50 / $15 | — | — |
| GPT-5.6 Luna | $1 / $6 | — | — |
On OpenAI’s side, the messaging plays on different ground: Sol claims state of the art on BrowseComp (92.2%) and OSWorld 2.0 (62.6%), the top of Artificial Analysis’s Coding Agent Index (80, or 2.8 points ahead of Fable 5), and 53.6 on “Agents’ Last Exam” — the famous “13.1 points ahead of Fable 5” from the press release.
See the pattern? Every vendor waves the benchmark it wins. Anthropic leads with SWE-bench; OpenAI with web agentics and value for money — Luna reportedly delivers about 24 benchmark points per API dollar versus 3.2 for Fable 5. Both are right… on their own turf.
The honesty section: four traps to know
1. The in-house score is not the independent score. Fable 5’s 80.3% on SWE-bench Pro was produced with Anthropic’s own harness — optimized prompts, custom tooling. The harness alone can be worth several points. The 95% on SWE-bench Verified, however, is confirmed by vals.ai, a third-party leaderboard. The reflex to build: look for the number replicated by a third party before quoting the press release.
2. Contamination. Models train on the web… where benchmarks are published. A model that “saw” the answers during training takes the exam with the answer key in its pocket. That’s why benchmarks constantly renew themselves (Verified → Pro → private test sets): once a task set lingers online too long, its predictive value erodes.
3. Saturation. When everyone hovers around 95%, the benchmark stops discriminating — the gap between 95 and 96% is measurement noise, not a ranking. MMLU met that fate; SWE-bench Verified is getting there. A saturated benchmark is a crash test every car passes: time to build a harder one.
4. The benchmark war. July’s juiciest episode: after Fable 5’s 80.3% on SWE-bench Pro was published (versus 64.6% for Sol), OpenAI released research estimating that “about 30% of SWE-bench Pro tasks are broken”. Maybe true! But a methodology critique that only shows up after a loss on that exact benchmark deserves your skepticism — in both directions.
How to read them intelligently
- Find the benchmark that looks like your workload. Doing .NET refactoring? SWE-bench multilingual says more than BrowseComp. Browser automation? The other way around.
- Divide by the price. A model 2 points behind at 5× cheaper often wins in production — that’s Luna and Terra’s whole argument, and it’s a fair one.
- Prefer independent leaderboards over press-release numbers, and check the harness and the pass@k.
- Benchmarks tell you which model to pick at the start — we covered that angle in choosing your model in Copilot. But to know whether your feature works, nothing replaces your own evals: a benchmark is a generic exam, your evals are the exam of your business.
In summary
| Concept | Takeaway |
|---|---|
| Benchmark | Frozen tasks + automatic metric + harness |
| pass@1 / pass@k | 1 attempt vs k attempts — compare like for like |
| In-house score | Vendor-optimized harness — needs replication |
| Contamination | The model may have seen the answer key |
| Saturation | At 95%+, gaps are noise |
- A benchmark is a standardized crash test: great for comparing, insufficient to guarantee your use case.
- Every vendor communicates on the ground where it wins — cross-check several benchmarks and third-party leaderboards.
- Score per dollar matters as much as the raw score.
- Benchmarks to choose the model, evals to validate your application.
Sources: Simon Willison on GPT-5.6, Vellum — Fable 5 benchmarks, vals.ai SWE-bench, MarkTechPost — Opus 5.
