Designing contamination-resistant benchmarks
Building tests that measure capability rather than memorization, and survive being trained on.
Every benchmark has a half-life. The day it becomes useful is roughly the day it starts leaking into training sets, and from then on it measures recall as much as capability. A year-old public benchmark is often closer to a vocabulary quiz than a test of reasoning, and the field’s response has mostly been to build a new one and start the clock again.
What contamination looks like from the outside
You usually see it as a suspiciously clean jump on the public score with nothing else moving, no lift on adjacent tasks, no change users can feel. The model didn’t get smarter between checkpoints; it read the answer key. Once test items are in the pretraining corpus, the benchmark stops measuring the broad ability you cared about and starts measuring how well the model memorized that particular set.
What actually helps
A few things, in rough order of effectiveness. Keep a private split you never publish, and treat the gap between public and private performance as your contamination gauge. Generate fresh instances procedurally, so no two evaluation runs reuse the same items. Grade by execution against hidden tests instead of string-matching a known answer. And where you can, design tasks whose answer depends on state a model could not have memorized, a specific repository at a specific commit, a randomized environment seed, source material published after the training cutoff. LiveBench is the cleanest public example of this discipline: it refreshes questions every month from recent papers, news, and competitions, and scores against objective ground truth rather than an LLM judge, precisely so that memorization buys you nothing.
Capability versus recall
Underneath all of that is one principle: a good item stays hard even for someone who has seen a thousand like it. If familiarity alone cracks it, you are testing memory. If it still demands the work no matter how many neighbors you’ve seen, you are testing capability. Most of the design effort goes into telling those two apart, because they look identical on a leaderboard.
And then there’s Goodhart
A benchmark is a proxy, and the moment a team starts optimizing the proxy instead of the underlying ability, the number climbs while the capability doesn’t. That failure has nothing to do with leakage, and no amount of held-out data fixes it. The defenses are mundane and worth doing anyway: plant a canary string and watch for it, rotate items, report the public-versus-fresh gap alongside the headline number, and keep asking whether the metric still tracks the thing you actually care about. The discipline is to retire a benchmark the moment it stops, rather than the moment everyone admits it has.