SFT Teaches Agents to Copy Moves. Agentic-DPO Teaches Them to Choose.

Supervised fine-tuning on expert trajectories looks like a reasonable training recipe for LLM agents. It is not. SFT treats every expert action as the next token to predict, which means the agent learns to reproduce sequences, not to evaluate competing actions at the states where decisions actually diverge. The agent becomes fluent at imitation and brittle at choice.

The failure mode is structural. At any decision point in a multi-turn agent task, there are plausible wrong actions alongside the correct one. SFT never shows the agent that contrast. It sees only the expert's path, never the adjacent paths that look reasonable but lead to failure. Production agent pipelines built on SFT alone inherit this blindspot directly: the agent performs well on trajectories that resemble training data and collapses when the task branches in a direction that requires genuine action selection rather than pattern recall.

Agentic-DPO fixes this without requiring online environment interaction. At each state in an expert trajectory, it samples one action from the current policy, treats that sampled action as a negative candidate, and contrasts it against the expert action using a DPO-style preference objective. The training signal is not "here is the correct sequence" but "at this state, this action is better than that action." The distinction sounds subtle. The effect on decision-point accuracy is not.

One design choice separates Agentic-DPO from naive offline preference learning: Policy-Preserving Augmentation (PPA). Multi-turn agent tasks involve two separable things, the policy (what the agent decides to do) and the schema (how that decision is formatted or expressed across different tools or APIs). Mixing both into preference learning creates a confounded signal: the model may learn to prefer one schema over another rather than one action over another. PPA renders the same latent trajectory under multiple schemas while holding the expert policy fixed, so the preference objective trains decision quality, not formatting convention.

The compute cost is low by design. No reward model. No full-trajectory student rollouts. No live environment calls during gradient steps. Only step-level sampling from the current policy at each expert state, which is the minimal rollout needed to generate a negative candidate.

Across StableToolBench, tau-bench retail, and Mind2Web, Agentic-DPO consistently improves over SFT baselines at multiple model scales. The headline result: tau-bench accuracy lifts from 21.7% to 41.4% on a 9B model, matching online GRPO trained under the same backbone. That match is the finding worth sitting with. GRPO requires full online environment interaction during training. Agentic-DPO produces equivalent accuracy with only step-level offline rollouts. For teams building multi-turn agent systems on top of open-weight models in the 7B-13B range, the takeaway is direct: preference training at the action level is now accessible without the infrastructure overhead of online RL.

We're thinking: We keep returning to the GRPO parity result, because it reframes a tradeoff that most teams treat as fixed. The assumption has been that online RL is expensive but necessary for real agent capability, while offline methods are cheap but leave accuracy on the table. Agentic-DPO's 41.4% matching online GRPO's number on tau-bench retail suggests the ceiling on offline training is higher than the field assumed, at least when the training signal is constructed at the state level rather than the trajectory level. The caveat worth naming: tau-bench retail is one environment, and the parity result may not hold on tasks with longer horizons or denser branching. But the architectural logic, that decision-point errors are the dominant failure mode and that state-conditioned contrast directly targets them, is general enough that teams should test this before committing to online RL infrastructure.

Key takeaways:

• Agentic-DPO converts each state in an expert trajectory into a preference pair by sampling one policy action as a negative, then applies a DPO objective at the step level rather than the sequence level. PPA decouples policy quality from schema formatting in the preference signal.
• On tau-bench retail, a 9B model trained with Agentic-DPO reaches 41.4% accuracy versus 21.7% for SFT, matching online GRPO with no environment interaction during gradient steps. Results hold across three benchmarks and multiple model scales, though long-horizon generalization beyond these benchmarks remains untested.
• Teams shipping multi-turn agent pipelines on open-weight models should run Agentic-DPO as a post-SFT training stage before investing in online RL infrastructure: the offline preference signal targets the exact decision-point failures that make production agents unreliable.

Source: Agentic-DPO: From Imitation to Agentic Policy Optimization on Expert Trajectories