Thinking Tokens Don't Deliberate on Safety: The Decision Is Already Made
New evidence shows reasoning models lock in refusal/compliance before visible thinking begins, with 0.84-0.95 AUROC predictability from the first token.
The assumption has been quiet but pervasive: if a model thinks before it answers, it will make better safety decisions. Thinking tokens give the model space to reason through whether a request crosses a line. That assumption turns out to be empirically wrong.
Across frontier open-weight reasoning models from the GPT-OSS, Qwen, OLMo, and Phi families, a trained probe on the first token's hidden representation predicts the final refusal or compliance outcome with 0.84 to 0.95 AUROC and roughly 88% balanced accuracy. The model has not produced a single visible thinking token yet. The decision is already made.
The mechanism here is closer to prefix completion than to deliberation. Once the model encodes the input, the hidden state at position zero carries enough signal to predict the output class with near-certainty. What follows in the thinking trace is not a reconsideration process. It is elaboration on a conclusion already reached. The final refusal or compliance outcome almost never changes after the first 20% of thinking tokens are generated. More telling: approximately 74% of text-level deliberations, the passages that look like the model is weighing options, occur after the response distribution has already committed to one side.
This is not a subtle statistical artifact. It is a structural property of how these models process safety-relevant inputs. The visible reasoning trace performs the appearance of deliberation without the function. For alignment researchers and teams shipping safety-critical pipelines, the takeaway is direct: treating thinking tokens as a safety mechanism adds latency and interpretive noise, not genuine reconsideration capacity.
The paper also tests whether existing interventions fix this. Both inference-time and training-based safety methods, designed with the explicit goal of inducing deliberation, largely produce over-refusal while suppressing the already-scarce deliberation signals. The interventions shift the decision threshold without changing the underlying architecture of how decisions form.
We're thinking: We find this result clarifying in a way that should make labs uncomfortable. The "deliberation equals alignment" story has been doing quiet work in how reasoning models are positioned, both internally and externally, as safety-forward by design. This paper shows that story is not supported by the mechanics. The thinking trace is not a safety scratchpad. It is post-hoc narration of a decision formed in the residual stream before any token is sampled. That means every safety evaluation framework that treats reasoning-model outputs as evidence of deliberative alignment is measuring the wrong thing. The more urgent question is whether any current training recipe can actually produce real safety deliberation, or whether the architecture itself forecloses it.
Key takeaways:
- Refusal and compliance outcomes in reasoning models are encoded in the first-token hidden representation, making the thinking trace a narration of a pre-formed decision rather than a deliberative process.
- A trained probe achieves 0.84 to 0.95 AUROC across multiple model families; outcomes stabilize after the first 20% of thinking tokens, and 74% of apparent text-level deliberations occur after the outcome is already fixed. Caveat: results are on open-weight models only; closed-weight frontier models may differ in ways not yet testable.
- Teams using reasoning models in safety-critical pipelines should audit whether their safety evaluations conflate the appearance of deliberation with actual reconsideration capacity, and should not treat longer thinking traces as evidence of stronger alignment.