Reasoning traces beat answer-only data
Why capturing the working, not just the result, is what lifts models on the hardest tasks.
A finished answer is the smallest, most lossy part of expert work. By the time a senior engineer commits a three-line fix, the hard part, working out which three lines, is already over and nowhere in the diff. Train a model on the diff alone and you are training it on the part that was easy.
The clearest evidence is in the gap between process and outcome
In 2023, OpenAI put numbers on something a lot of practitioners already suspected. In “Let’s Verify Step by Step,” a reward model trained on the individual steps of a solution beat one trained only on whether the final answer was correct, and not by a rounding error. On the MATH benchmark the process-supervised model pulled clearly ahead, and the team went to the trouble of releasing PRM800K, eight hundred thousand step-level human judgments, precisely because the per-step signal was where the value turned out to live. The result is about math, but the lesson is not: the supervision you can attach to the working is richer than the supervision you can attach to the answer.
Why answer-only data plateaus
Show a model enough (problem, final answer) pairs and it gets very good at producing outputs that look right. On problems whose shape it has seen, that is frequently enough to pass. On a genuinely new one it has nothing to fall back on, because you never showed it a procedure, only destinations. It can pattern-match its way to a plausible answer, but it cannot run the search that would have found the right one.
What a trace actually contains
The trace is that search. It carries the misread that got caught two steps later, the test that failed and what that failure ruled out, the moment someone decided to refactor instead of patch and the reason they gave. Those decisions are the part that transfers to the next problem. They are also, not coincidentally, the part that no amount of scraped text contains, because nobody writes them down.
Capture it as it happens, don’t reconstruct it after
There is a tempting shortcut here worth naming so you can refuse it: ask the expert to write up how they solved it afterward. Don’t. Post-hoc explanations are clean stories, and the clean story is rarely what happened, people reconstruct a tidy path they did not take. The signal is in instrumenting the work while it unfolds, dead ends included. This is also why the recent wave of RL-for-reasoning works at all: in the verifiable-rewards recipe behind models like DeepSeek-R1, the model is rewarded for producing a trace that verifiably reaches the answer, not just the answer. Whether that teaches genuinely new reasoning or mostly sharpens paths already latent in the base model is still argued, a 2025 study found RLVR largely reweights what the base model could already sample, but either way, the trace, not the answer, is the unit that carries the signal.
Where to spend the budget
None of this means traces everywhere. They are more expensive to collect and noisier to clean, and for lookup-style tasks the extra structure buys little. The lift concentrates exactly where you’d expect: long-horizon work, debugging, anything that requires noticing you’re wrong and recovering. Put the trace budget where the path is the product, and use cheaper answer-only data where it isn’t.