Run RL on the Small Model, Transfer the Gains Up: Direct-OPD

Post-training with reinforcement learning from verifiable rewards works. The problem is that it scales badly: every new model needs its own rollout budget, and rollouts on large models are expensive. The assumption baked into most RLVR pipelines is that the training signal is inseparable from the model generating it. Direct-OPD breaks that assumption.

The method starts by running RL on a small, cheap model until its policy shifts. That shift, not the final policy, is what gets transferred. Concretely, Direct-OPD computes the log-ratio between the post-RL teacher and its own pre-RL reference checkpoint. That ratio encodes which actions RL made the weak model more or less likely to take. It then applies that signal on the stronger student's own on-policy states, without training an explicit reward model or running sparse-reward RL on the target model at all.

Think of it as extracting the direction of improvement rather than copying the destination. A small model trained on RL ends up somewhere better than where it started, but that somewhere is still limited by the small model's capacity. The log-ratio between before and after captures the direction of the journey, not the endpoint. A stronger model, starting from a higher baseline, can follow that same directional signal and arrive somewhere better still. This also enables sequential composition: multiple policy shifts from different RL runs can be stacked, letting teams accumulate reasoning improvements without re-running the full pipeline each time.

Direct-OPD lifts Qwen3-1.7B from 48.3% to 62.4% on AIME 2024. That 14.1-point gain arrives in 4 hours on 8 A100 GPUs. It outperforms step-matched direct RL on the target model, meaning the transferred signal is not just cheaper, it is more effective per step. For teams running post-training at scale, the takeaway is direct: the most expensive part of RLVR, generating rollouts on the large model, can be skipped entirely without sacrificing the quality of the resulting policy shift.

We're thinking: We find the sequential composition result the most consequential detail here. If policy shifts are modular and stackable, labs could maintain a library of RL-induced shifts trained on small models across different reasoning domains and compose them onto a single strong base, rather than re-running expensive RL every time a new capability is needed. That reframes post-training budgets away from "run RL on the frontier model" toward "run many cheap RL experiments on small models and transfer selectively." The open question is where the ceiling is: at some gap between teacher and student capacity, the directional signal may no longer generalize cleanly, and the paper does not yet characterize that boundary.

Key takeaways:

• Direct-OPD transfers a weak model's RL-induced policy shift via a log-ratio implicit reward applied on the strong model's own on-policy states, bypassing both reward model training and target-model rollout generation.
• Qwen3-1.7B gains 14.1 points on AIME 2024 (48.3% to 62.4%) in 4 hours on 8 A100s, outperforming step-matched direct RL; caveat is that generalization across large teacher-student capacity gaps remains uncharacterized.
• Teams running RLVR post-training on large models should pilot Direct-OPD by training RL only on a smaller model in the same family and distilling the policy shift, treating rollout compute on the large model as a cost to eliminate rather than a given.

Source: Weak-to-Strong Generalization via Direct On-Policy Distillation