Item-level analysis of frontier-model benchmarks
Are our benchmarks still telling models apart?
Of 24,045 scored questions, 38% give every model the same result. We show which benchmarks still separate the 13 models, and at which ability levels.
01 / Signal inventory
Item count overstates how many questions still separate models.
We call an item resolving when model outcomes differ and the fitted item has neither a low-discrimination nor a high-uncertainty flag. A saturated item gives all 13 models the same outcome.
These rates describe this model roster and scoring setup. A saturated item may still verify minimum competence, but it cannot distinguish the models compared here.
02 / Shared signal
Seven benchmark scores mostly reproduce the same model ordering.
For the 13 models evaluated on all seven benchmarks, the first principal component explains 89% of standardized benchmark-level ability variance.
Pairwise correlations between benchmark ability estimates range from 0.73 to 0.96. High agreement leaves little benchmark-level variation independent of the common ranking.
03 / Ability coverage
Each benchmark separates models over a different ability range.
On the linked θ scale, information peaks at θ=-3.62 for GSM8K, θ=-2.26 for ARC-Challenge, and θ=+2.65 for AIME 2024.
All curves use the same linked θ axis. Each curve is normalized independently, so compare its horizontal location and shape, not its height.
Inspect the evidence
Inspect the items that still separate models.
Use the leaderboard to compare model scores. Use the IRT analysis to inspect each item’s difficulty, discrimination, uncertainty, and contribution to benchmark information.