We only ship the tasks that beat your model.
Every task is run by frontier agents before it leaves here. If they solve it, we throw it away. Difficulty on this site is a measurement, not an adjective.
What survives. Every week thousands of tasks arrive, from credentialed experts across six continents. Each is pinned, built and made reproducible. Then frontier agents try to destroy them, and most of the batch does not survive. What is left is the only thing worth training on.
The bench
The bench is never asleep.
Six hubs, spread far enough apart that a normal working day in each one overlaps into continuous cover. Not a claim about effort — just what these longitudes add up to. Hover a row for what that hub is for.
- Frontier training & compute — The deepest pools of compute, capital and research talent, and where most frontier models are trained.
- Where deep learning grew up — Toronto and Montreal nurtured modern deep learning long before it was fashionable.
- Foundational research — A century-deep bench in mathematics and computer science feeding world-class labs.
- Sovereign compute — Sovereign investment behind some of the largest compute build-outs outside the US and China.
- Where AI gets engineered — A vast software talent pool, and a linguistic proving ground for AI that has to work for everyone.
- Southeast Asia hub — Research institutes, capital and multinational R&D packed into one highly connected city.
Union of all six windows 24 / 24 hours · no gap
Loading local times
A case file · Mohit Labs
Most training data is bought by the ton and never tested. Ours has to survive a trial first — and what you just watched is the part that made it through.
Read the fileFollow a single real task from the bug that produced it to the dataset it ended up in, and see what happened when three frontier agents tried to kill it.
- Subject
- apache/arrow #41207
- Language
- C++
- Filed by
- Systems eng · 9 yrs
- Outcome
- Accepted
Chapter one
It starts with something actually breaking.
Not a puzzle invented for a benchmark. A real defect, found by someone already deep in the code who noticed the arithmetic was wrong past the first chunk.
$ ctest -R chunked_array_slice Start 1: ChunkedArray.SliceBasic ............ Passed Start 2: ChunkedArray.SliceEmpty ........... Passed Start 3: ChunkedArray.SliceAcrossChunks .... FAILED Start 4: ChunkedArray.SliceOffsetRebase .... FAILED Start 5: ChunkedArray.SliceZeroLength ...... FAILED expected items[20:25] actual items[20:23] <- offset never rebased 11/14 passed · 3 failed
Chapter two
The fix is the easy part. The reasoning is the product.
Anyone can commit three lines. What we capture is the working — which branch was ruled out, why, and the check that proved it. Then it is pinned so it reproduces exactly, forever.
- Base commit
- a4f9c21e8b7d
- Environment
- Pinned Dockerfile — builds clean from scratch
- Failing tests
- 3 must fail before the fix, and pass after it
- Reference fix
- 812 lines across 7 files
- Captured reasoning
- Ruled out the empty-slice branch — it hides the bug rather than fixing it
- Leakage scan
- Instruction names no identifier from the solution
Chapter three
Then we try to destroy it.
Three frontier agents, eight independent attempts each, in the same sandbox a customer would run. Twenty-four chances to make this task worthless. Every filled square is an attempt the model won.
2 solves in 24 attempts
Alpha stalled rebasing chunk offsets · Beta regressed zero-length chunks · Gamma quit after two failed cycles
Filed the same week · did not survive
- Cut
CSV quoting round-trip · pandas— solved 8/8, nothing left to teach - Cut
Off-by-one in paginator · django— solved 7/8 - Cut
Null deref in config loader— instruction leaked the function name - Cut
Race in scheduler · internal— environment would not build twice the same way
Chapter four
It survived. That is the whole qualification.
Each agent gets eight attempts. This task went 0, 1 and 1 — a median of one solve per eight, which sits at the bottom of the window: hard enough to carry signal, not so hard it is broken. A task landing outside goes back to its author with the reason attached.
1 / 8Median · inside the window · kept
From here the same recorded session ships as a trajectory, an RL environment, an eval item or a preference pair — nothing is re-annotated afterwards, and the solve rate above travels with every one of them. See the full surface →
What we make
That one task ships in sixteen forms.
Post-training data, agentic environments and benchmark construction are usually three different vendors. They come out of one pipeline here, from the same recorded sessions — which is why the solve rate travels with them.
- Post-trainingSupervised fine-tuning · Preference & RLHF · Code generation · Multimodal04
- Reward & RLRubric & verifier grading · Tool-calling environments · Durable RL environments03
- AgenticAgent trajectories · Computer & browser use · Long-horizon tasks03
- EvaluationBenchmarks & evals · Deep research tasks · Failure & loss analysis03
- DeliveryOff-the-shelf datasets · Custom evals & datasets · Professional domains03
Research & notes
How we think about data and evaluation.
All notes →Reasoning traces beat answer-only data
Why capturing the working, not just the result, is what lifts models on the hardest tasks.
May 2026 · 8 min read
Designing contamination-resistant benchmarks
Building tests that measure capability rather than memorization, and survive being trained on.
Apr 2026 · 9 min read
What makes an RL environment trainable
The properties that separate a useful, well-shaped environment from a brittle one.
Apr 2026 · 7 min read
The 11-criterion review bar, explained
How every item earns its place in a dataset, and what we reject.
Mar 2026 · 6 min read
FAQ
For AI teams, answered.
Still have a question?
What actually makes a task “hard” here?
Not an author’s opinion. Every task is run by frontier agents before it ships and the solve rate is recorded. Solve it too often and it is rejected — a task a capable model can already complete teaches your model nothing. Difficulty is a measured property of the task, and it travels with the data.
How does an engagement work?
We scope the capability or evaluation you need, run a small pilot to calibrate quality against your models, then scale. You can commission bespoke work or pull from review-cleared off-the-shelf datasets.
Who are the experts?
Credentialed practitioners with real on-the-job experience in their field, not generic annotators. Every contribution clears automated validation and layered human review before delivery, and contributors progress through tiers gated on accepted work.
What formats do you deliver?
SFT, preference and RLHF data, agent and computer-use trajectories, RL environments, code, multimodal data, and evaluation suites — in your schema, ready to train on.
How do you handle contamination?
Originality is a gating check, not a spot audit: every submission is compared against the existing corpus and rejected when it is too close. Tasks are built on pinned commits with held-out tests, and the fairness stage scans instructions for leaked identifiers before anything is accepted.
Who owns the data, and is it secure?
You own the deliverables. We work under NDA, isolate engagements, and scope handling to your security requirements.
The end of the file
Now tell us where your model falls short.
Bring the capability you are trying to move. We will build the tasks that defeat it, prove they do, and report the number.