A real build, scored by what happened in the repo.
We run the real agent CLIs (Claude Code, Codex) against real repos and measure what they do, not what they say. Detection is deterministic, never an LLM grading an LLM.
Data through 2026 Q2 · last updated July 2026
How a run works.
- 01 · ask
A real task, worded the way developers word it. Each task runs several different ways so no single phrasing decides a number.
- 02 · build
The agent works in a real repo, fresh projects and existing codebases alike. Real installs, real code, real npm run build.
- 03 · record
We read what actually happened in the repo: what was installed, what the code uses, whether the build passed. Nothing is inferred.
- 04 · classify
A classifier written before the runs scores every outcome, so nobody grades a result after seeing it, and no LLM judges another LLM.
- 05 · gate
Every share ships with n; anchor tables and the public dataset carry 95% Wilson intervals. Below n=30 a cell is underpowered, not reportable.
Two layers.
Layer 1 · Recommendation probes
The agent is asked what it would use; no code is written. High sample counts. Answers "who gets named" across many wordings of the same task.
Layer 2 · Build journeys
The agent actually builds, in fresh projects and in existing codebases, including ones where a competitor is already installed. Answers "what really happens," including the gap between what agents say and do.
Four metrics, in funnel order.
- 01 · named
Agent Reach
How often agents name you when asked to build. Tracked at every model release.
- 02 · installed
First-Install Rate
Of the times you're named, how often the agent actually installs you.
- 03 · compiled
Build Success
When installed, whether the project compiles. Measures compilation, not runtime correctness.
- 04 · kept
Refactor Retention
Whether your integration survives a “modernize this” pass or gets swapped out.
Honest by construction.
Shares carry their evidence
Category-share and model-comparison tables include sample sizes and 95% Wilson intervals; other exploratory findings are labeled separately.
A 30-run reportability floor
Below n=30 a cell is underpowered, not reportable. We don't headline thin data.
Validated detection
Forty archived runs re-labeled blind by five independent LLM raters (shown only the code, never our label): vendor detection 95% precision, 100% recall (18/18); overall outcome accuracy 87.5% (35/40) — the known weak spot is DIY detection. The recommendation parser: 97% recall, 86% precision. The eval set is versioned with the harness and ships with the first citable release. We test the tests.
What we don't claim.
Compiled means compiled
Build Success measures whether the project compiles, not whether it behaves correctly in production.
Agents, not humans
We measure what coding agents do, not what human developers prefer. Those are different populations; ours is the one growing.
Controlled, not yet a citable release
Current figures come from controlled runs and always carry n; below n=30 a cell is underpowered. A citable release is a stricter, versioned artifact under a frozen protocol, and we have not shipped one yet.
Publication policy: public releases identify category leaders and aggregate behavior. Detailed vendor-level results stay private until AgentRank introduces a complete-table release under predefined eligibility and publication rules. Customer or prospect status never affects publication, in either direction.
Prompts are versioned and sampled with a fixed seed, and every run records a manifest of exactly how and when it was produced, down to the model and agent version. Since July 2026 the Claude Code reasoning-effort setting is pinned: a batch refuses to start if the runner's setting differs from the pinned level, and the level is recorded in every run's manifest. Today the public artifact is a summary dataset; the full release bundle ships with the first citable release, and until then this page says what is public and what is not. The seed governs prompt sampling and task assignment, not model output: hosted agents are not deterministic, so the reproducible object is the protocol and evaluation environment, not each generated trajectory.
Terms.
- top-pick share
- Of all runs in a category, the share where a tool was the agent's first choice. Different from being named: an agent can mention a tool constantly and never pick it.
- switch graph
- When an agent considers a tool but installs another, the switch graph records who won instead and how often. Computed per vendor; lives in the private scorecard.
- reportable
- A figure backed by at least 30 runs, published with its sample size and a 95% Wilson confidence interval.
- underpowered (formerly directional)
- A figure below the n=30 reportable bar. Useful for direction; never presented as a published benchmark number.
- AgentRank Score
- A 0-100 composite of the four metrics, computed over reportable components only and shown with its confidence band.
Common questions.
How does AgentRank benchmark AI coding agents?
We run the real agent CLIs (Claude Code, Codex) against real repos: real installs, real code, real builds. Detection reads what actually happened in the repo, and a classifier written before the runs scores every outcome. Every figure ships with its sample size and a 95% Wilson interval.
Does an LLM judge the results?
No. Outcomes are scored by deterministic detection plus a pre-specified classifier, never an LLM grading an LLM. Blind raters put vendor detection at 95% precision and 100% recall, and the recommendation parser at 97% recall.
When is a number reportable?
Four statuses. Underpowered: below 30 runs, never a headline. Reportable: at least 30 runs, shown with n. Exploratory release: controlled runs published before the frozen citable protocol (all current figures). Citable release: a frozen, versioned, hash-committed bundle (none shipped yet). Anchor tables and the public dataset carry 95% Wilson intervals.
Which agents and models does AgentRank measure?
Claude Code and Codex today, with every metric split by model, because standing differs sharply between models (the same tasks compiled 89% on Claude Opus and 73% on Sonnet in Q2 2026). We re-run at model releases.
Related reading: Wilson score interval · On the impact of AGENTS.md files (arXiv, 2026) · Mintlify: agents are half of docs traffic
The method is public so the numbers can be checked. See what they say.