The Real Benchmark for AI Isn't Tokens—It's What Ships
The practice of measuring the performance of AI coding tools by token usage (processing units that split text into small chunks) may be about to change. In a piece published under the name Aphan Sied on November 2, 2026, MIT Technology Review argued that the performance of AI coding agents—AI systems that autonomously carry out coding tasks—should be measured not by token consumption but by actual shipped software.
When a developer hands a task to a coding agent, unlike a simple question-and-answer exchange, the agent repeatedly plans, generates code, tests, and fixes errors, with a single task consuming 10 to 50 times more tokens than a basic chat prompt. The more useful agents become, the more frequently they are called—and under per-token pricing, a paradox emerges: the tasks that create the most value end up being the most expensive.
The financial pressure is already a reality. Uber exhausted its 2026 AI budget by April after expanding its use of Claude Code. GitHub Copilot switched its billing model from flat-rate subscriptions to usage-based credits. Gartner predicts that AI coding token costs will exceed the average developer's annual salary by 2028.
In response, companies are turning to rationing-style controls: usage caps, seat limits, and approval requirements for agent runs. However, the opinion piece pointed out that blanket restrictions also constrain workflows that generate value. Per-token cost measures consumption, not outcomes. The piece also criticized token cost meters for failing to distinguish waste, such as retries or infinite loops, from work that leads to a release—pushing teams to optimize the wrong metric.
The alternative metric proposed is cost per merged pull request: total AI-assisted development costs divided by the number of changes that pass review and are merged. It also recommended tracking rework and defects alongside this metric.
The piece argued that expensive inference should be reserved for tasks that truly require it, while lowering the unit cost of repetitive work. Not every stage of agent coding needs a top-tier model. The approach: use frontier models for difficult planning and novel reasoning, and assign code generation, test writing, and iterative refactoring to self-hosted open-weight models. Gateways can route requests based on task complexity, data sensitivity, cost, and latency—ensuring premium inference is used only when it is justified.
The piece presented a shift away from token savings toward measurement centered on merges and shipped software as the new standard for AI coding cost management. With its publication in MIT Technology Review, industry discussion on how to manage AI coding costs is expected to continue.
