The 11-criterion review bar, explained
How every item earns its place in a dataset, and what we reject.
A dataset is only as good as the worst item that cleared review. That single sentence is why our bar has eleven criteria and why all of them have to pass, not most. What follows is what each group is actually checking for, and the kind of thing it throws out.
The execution criteria
Verifiable and behavioral-verification rule out anything decided by taste: if a test can’t run code and return pass or fail, it doesn’t ship. We reject “tests” that grep the source for a function name, that checks how something was written, not what it does, and a model can satisfy it without solving anything. This is the same instinct that makes verifiable-reward RL and SWE-bench-style hidden tests work: judge the outcome by running it, not by inspecting it.
The specification criteria
Well-specified means exactly one reasonable reading; solvable means a reference solution exists and actually passes. The most common rejection here is an instruction that quietly names the approach. The moment a prompt says “use a two-pointer scan,” it has stopped measuring problem-solving and started measuring reading comprehension, and a model trained on it learns to follow recipes rather than find them.
The difficulty and fairness criteria
Genuinely-difficult keeps items in band, neither trivial nor impossible. Fair kills traps that punish a correct solution for being the wrong shape. Anti-cheat-robust means you can’t pass by gaming the check rather than doing the task. Deterministic means the same input yields the same verdict on every run. A surprising number of plausible tasks fail “fair”: they accept the intended answer and one specific style, and quietly reject other solutions that are just as correct.
Why none of it is optional
Each criterion is on the list because we once shipped something that failed it, and a model trained on that item learned the wrong lesson, to mimic a style, to exploit a verifier, to memorize a quirk. So every submission comes back accepted, revised, or rejected, always with the reason attached. A verdict without a reason teaches no one anything, and the reason is half the product.