FIM Mid-Training Adds +5.4 on SWE-Bench Without Changing the Agent Pipeline

Standard left-to-right pretraining on code teaches a model to produce function calls. It never teaches the model to reason from the callee's perspective: receiving a return value from somewhere else and continuing downstream logic correctly. That asymmetry is not a model size problem. It is a training objective problem, and it shows up every time a coding agent has to integrate a tool's output into ongoing reasoning.

The fix is structural. The action-observation-continuation loop that defines every coding agent, where the agent calls a tool, receives a result, and keeps reasoning, is isomorphic to a function call site in ordinary source code. A caller binds arguments, a callee computes and returns a value from outside the current context, and downstream code consumes that value. This conditioning structure exists at internet scale in public repositories. Function-aware fill-in-the-middle (FIM) mid-training makes that structure explicit as a training signal.

The method masks functions selected through program dependency graph analysis, applying a double criterion: the masked function must be complex enough to carry real semantic weight, and its behavior must be inferable from surrounding context. That second criterion is what separates this from naive FIM. Masking random spans teaches a model to fill gaps. Masking dependency-graph-selected functions with inferable behavior teaches the model to reconstruct a callee's contribution from the caller's context and the downstream consumer, which is exactly the skill a coding agent needs when a tool returns an unexpected value mid-trajectory.

Mid-training runs on a 2.6B-token corpus drawn from 968 GitHub repositories, decontaminated against evaluation benchmarks, and uses only Python code. The base models are Qwen2.5-Coder-Instruct at 7B and 14B, plus Qwen3-8B. After mid-training, existing agentic post-training pipelines (R2E-Gym, SWE-Smith, SWE-Lego) apply without modification.

SWE-Bench-Verified improves by +2.8 and +3.0 at 7B and 14B respectively, and by +3.2 on Qwen3-8B. SWE-Bench-Lite gains are larger: +3.7, +4.0, and +5.4 on the same three models. The gains hold across all three post-training pipelines and on the non-Qwen2.5 base, which confirms the effect is not an artifact of a specific recipe or architecture family. Equally notable: mid-training on Python-only code mitigates the capability erosion that agentic post-training typically inflicts on general coding benchmarks (LiveCodeBench) and non-coding tool-use tasks (tau-bench, BFCL). The function-call inductive bias survives post-training and transfers. For teams shipping coding agents at the 7B-14B scale, the takeaway is direct: inserting a FIM mid-training stage before your existing agentic post-training pipeline is now a concrete, reproducible way to lift agent performance without touching the pipeline itself.

We're thinking: We find the capability-preservation result more significant than the headline SWE-Bench numbers. Agentic post-training is well-documented to degrade general coding ability, and most teams treat that as an acceptable tradeoff. This paper shows the tradeoff is partly an artifact of the base model's training objective, not an inherent cost of specialization. If FIM mid-training installs a function-call inductive bias that survives post-training and generalizes to non-Python tool-use benchmarks from a Python-only corpus, then the training recipe is doing more architectural work than the model size. Teams evaluating whether to scale from 7B to 14B for agent tasks should run this mid-training stage first. The delta may close the gap more cheaply than a model upgrade.

Key takeaways:

• Function-aware FIM mid-training installs a callee-perspective inductive bias by masking dependency-graph-selected functions under a complexity-and-inferability criterion, making the training objective structurally match the action-observation-continuation loop of coding agents.
• SWE-Bench-Lite improves by +3.7 to +5.4 across three model variants and three post-training pipelines; the effect holds on a non-Qwen2.5 base and partially transfers to non-coding tool-use benchmarks, though the corpus is Python-only and cross-language generalization is not directly tested.
• Teams running agentic post-training on 7B-14B coding models should insert a FIM mid-training stage before their existing pipeline, using dependency-graph-guided function masking, before considering a model size upgrade.

Source: Function-Aware Fill-in-the-Middle as Mid-Training for Coding Agent Foundation Models