The most misunderstood graph in AI just got a major update
AI is still advancing rapidly. According to a report from METR, an AI evaluation nonprofit, AI performance is improving steeply (exponentially), but on tasks requiring human-like judgment, it has shown weaker results than in coding. According to the original plot, the length of tasks that frontier models can successfully complete has been doubling every four months. AI's performance has improved dramatically especially in "verifiable" domains like coding and math, where success can be automatically checked.
In this update, the bundle of tasks has been expanded. Newly added are tasks estimated to take humans far longer to complete, as well as open-ended tasks with no fixed correct answer. It is difficult to apply reinforcement learning to tasks whose success is hard to verify automatically. Reinforcement learning works by having a system attempt the same task many times, strengthening the neural pathways that contributed to success and weakening those that led to failure. Because evaluation was difficult, METR's original time-horizon plot did not include tasks with less clearly defined success criteria.
This update shows both the exponential trend in AI capabilities and the fact that performance lags on tasks where success is hard to judge.
