An exam measures what it asks. When the exam becomes the target, the model learns the exam — and we mistake a memorised syllabus for understanding. The failure is old; the scale is new.
Transfer as the unit of merit
We argue that the quantity worth measuring is not accuracy on a held-out split drawn from the training distribution, but accuracy on a task whose form the model has never seen. Same underlying skill, unfamiliar clothing.
A protocol
For each capability, we author two families of tasks that share a latent skill but differ in surface structure. A model trains on the first and is scored only on the second. The gap between in-form and out-of-form accuracy is the transfer penalty — and it is far more predictive of real deployment behaviour than either score alone.