What a resolved issue actually costs
Updated: September 27, 2026
A benchmark that evaluates models on private, real-world enterprise repositories launched in September 2026 and drew heavy discussion. The premise matters more than the leaderboard: public benchmark suites are public, and a model that has seen them is not being tested on the code your engineers actually maintain.
Why a pass rate is not a finance number
A pass rate answers a capability question. It cannot answer a budget question, because a budget question needs three inputs at once:
- Success rate on tasks that resemble your backlog.
- Tokens per attempt, including the failures.
- Attempt multiplier — did it need a retry, a plan revision, a diff-and-rethink loop?
Multiply those and you get cost per resolved outcome. Two models a few points apart can differ threefold on that number once retries, plan revisions and rejected diffs are counted. The point spread is the least important part of the result.
Worked example
Take a 40-issue scope. Model A resolves 22 in a single attempt at 60k input and 8k output. Model B resolves 26 with a 1.4x retry multiplier at 90k input and 14k output. At a reported $4/$20 flagship rate with $0.20 cache reads:
| Model | Resolved | Cost per outcome | Total |
|---|---|---|---|
| A | 22 | $0.32 | $7.04 | B | 26 | $0.61 | $15.86 |
Model B costs 2.3x the total for 18% more resolved issues. Scaled to 4,000 issues a month that is roughly $880 more for 18% more throughput. Whether that is a good trade depends on what a missed issue costs, which is a business input no benchmark supplies.
The four numbers to instrument
- Cost per resolved outcome, not per request. The denominator is merged outcomes, not calls.
- Retry multiplier per model. A model at 1.4x is already a third cheaper than its list price before anything else is counted.
- Tokens per accepted change, including plan and self-review tokens the diff never shows.
- Human review minutes. A 12% cheaper model that takes 20% longer to review is more expensive. This is the number most often omitted, and it is the one finance cares about.
Bottom line
Until you run a private-codebase benchmark on your own repos, treat every model comparison as a cost-per-outcome hypothesis and price it with your own token counts and your own review hours. The pass rate is marketing; the resolved outcome is the budget.