Your RL Training Loss Is Optimizing the Wrong Policy
Training-inference probability mismatch silently corrupts LLM RL post-training. MIPU fixes the actual objective: the inference policy, not the training loss.
Every RL post-training pipeline assumes the same thing: improve the training policy, and the inference policy improves with it. That assumption is wrong, and the gap it creates is one of the underappreciated root causes of RL instability in LLM fine-tuning.
The structural problem is this: modern LLM training stacks use separate engines for generation and gradient updates. The inference engine runs in lower precision for throughput; the training engine runs in higher precision for stability. Even when both engines share synchronized weights, they assign measurably different probabilities to the same token sequence. That probability gap is not a rounding error. It is a persistent, always-on source of off-policy noise that never disappears no matter how carefully the training loop is tuned.
Prior work has tried to close this gap by correcting for off-policyness inside the training objective, using importance sampling ratios or clipping tricks borrowed from classic RL. Those fixes treat the symptom. They stabilize the training policy. But the training policy is not what gets deployed. The inference policy is. A gradient step that looks like an improvement from the training engine's perspective can degrade the actual inference policy, and no existing loss function catches that.
MIPU, the framework introduced here, reframes the optimization target entirely. Instead of minimizing a training-side loss, it asks a different question at each update step: does this candidate policy update actually improve the inference policy? The framework works in two stages. First, it constructs a set of candidate updates referenced against the sampler distribution, the policy that generated the rollouts. Second, it evaluates each candidate using an inference-side gap proxy, a lightweight measure of whether the inference engine's probability assignments move in the right direction. Updates that pass get accepted; updates that would degrade the inference policy get filtered out. Think of it as a gating layer between the optimizer and the deployed model, one that only lets through changes that matter where it counts.
The design contrast with standard PPO or GRPO variants is direct. Those methods assume the training engine is a faithful proxy for the inference engine and optimize accordingly. MIPU treats the two engines as structurally distinct and builds the acceptance criterion around that distinction rather than papering over it.
Experiments across two model scales under high-mismatch conditions show MIPU improves average reasoning performance and training stability compared to baselines that address off-policyness at the training level only. The gains hold across the reasoning benchmarks tested, and the stability improvement is visible in training curves that show fewer collapse events. For ML engineers running RL post-training pipelines, the takeaway is direct: if your training loss is converging but your eval performance is erratic or declining, the inference-training probability gap is a likely culprit, and patching the loss function will not fix it.
We're thinking: We find the framing here more significant than the benchmark numbers alone suggest. The field has spent considerable effort on training-side fixes for RL instability: better clipping, tighter KL constraints, curriculum tricks. MIPU's diagnosis points somewhere different. The inference engine is not a detail to abstract away; it is the actual deployment artifact, and any optimization objective that ignores it is misspecified by construction. The practical implication for teams is uncomfortable: a well-tuned training run with a healthy loss curve can still be silently degrading the model that ships. That means eval on the inference engine, not just training metrics, needs to be part of the RL training loop itself, not just a post-hoc check.
Key takeaways:
- MIPU separates the optimization target into two stages: constructing sampler-referenced candidate updates, then filtering them through an inference-side gap proxy that directly measures inference policy improvement rather than training loss reduction.
- Across two model scales under high training-inference mismatch, MIPU improves average reasoning performance and training stability over baselines; the caveat is that results are scoped to reasoning tasks and high-mismatch configurations, so gains under low-mismatch setups remain untested.
- Teams running RL post-training for reasoning or instruction-following should instrument their inference engine's probability assignments during training, not just monitor training loss, and treat divergence between the two as a signal to filter or reject candidate updates.