OPID: Dense Token Supervision from a Model's Own Rollouts, No External Skill Bank

Outcome-based RL for language agents has a well-known weakness: trajectory-level rewards tell a model whether it succeeded, not which of the dozens of intermediate decisions actually caused the success or failure. The standard fix is to import dense supervision from outside, through retrieved skill memories or privileged context stores. That fix introduces a new problem. External skill banks are expensive to maintain and, critically, they reflect a different state distribution than the one the current policy actually visits during training.

OPID sidesteps the retrieval problem entirely by mining supervision from the trajectories the model already generated. After each completed episode, OPID extracts two tiers of hindsight signal from that same rollout. Episode-level skills capture global workflow patterns or failure-avoidance rules visible across the full trajectory. Step-level skills capture the local decision logic at specific critical timesteps, the moments where the agent's choice most determined the outcome. Think of it as annotating a chess game after the fact: you can mark the one move that lost the endgame, and you can also note the broader positional strategy that constrained every move before it.

The routing mechanism is what makes this practical. A critical-first selector identifies timesteps where the decision was high-stakes, and injects step-level skill context at those points. Everywhere else, the episode-level skill serves as default guidance. That selected skill gets appended to the interaction history, and the old policy then re-scores its own sampled response under both the original context and the skill-augmented context. The log-probability shift between those two scorings becomes a token-level self-distillation advantage. That advantage is added to the standard outcome advantage, so RL remains the primary optimization objective and the dense signal supplements rather than replaces it. Distribution mismatch disappears because the skill was extracted from the same policy that is being updated.

On ALFWorld, WebShop, and Search-based QA, OPID improves over outcome-only RL baselines and over existing skill-distillation methods across agent performance, sample efficiency, and robustness metrics. For teams training agentic LLMs on multi-turn interactive tasks, the takeaway is direct: you can get dense token-level supervision without building or maintaining any external knowledge infrastructure, because your own completed rollouts already contain it.

We're thinking: We find the distribution-matching argument more significant than the benchmark numbers alone suggest. Most skill-conditioned distillation pipelines fail quietly in production not because the skills are wrong in isolation, but because retrieved skills were written under a different policy's state distribution and no longer match what the current model actually encounters. OPID dissolves that mismatch structurally, not through better retrieval. The implication worth watching: if on-policy hindsight extraction generalizes to longer-horizon tasks, it may make the entire class of external skill-memory architectures unnecessary for agentic fine-tuning, which would meaningfully reduce the operational surface area of production training pipelines.

Key takeaways:

• OPID extracts hierarchical hindsight skills directly from completed on-policy trajectories, using a critical-first routing mechanism to inject step-level or episode-level signals at the right timesteps, then converts the log-probability shift under skill-augmented context into a token-level distillation advantage added to the outcome reward.
• Across ALFWorld, WebShop, and Search-based QA, OPID outperforms both outcome-only RL and existing skill-distillation baselines on performance, sample efficiency, and robustness; the primary caveat is that evaluation remains on relatively short-horizon benchmarks, leaving open how the approach scales to tasks with hundreds of decision steps.
• Teams training agentic LLMs with outcome-based RL should test OPID's on-policy extraction loop as a drop-in supplement before investing in external skill banks or retrieval infrastructure.

Source: OPID: On-Policy Skill Distillation for Agentic Reinforcement Learning