Your Agent Benchmark Score Is a Harness Score in Disguise

Agent benchmarks report whether the task succeeded. That number feels clean and comparable. The problem is that success can survive harness changes that quietly corrupt the beliefs driving every decision along the way.

The standard assumption in agent evaluation is that if the base model, the task, and the environment are fixed, the harness is just plumbing. Swap SWE-bench's scaffolding for your own, and the score tells you something about the model. This paper breaks that assumption directly: the harness controls what the agent sees, which actions it can attempt, which failures get repaired, and which evidence gets pruned. Those choices reshape belief trajectories even when the terminal outcome stays the same.

The mechanism works through two separable effects. The first is an arrival term: the immediate interface shift caused by a new harness changes the agent's starting beliefs about progress, risk, and recoverability before any multi-step reasoning begins. The second is a growth term: as the agent runs forward through K steps, harness-induced belief gaps compound, so the divergence at step 5 is not the same shape as the divergence at step 1. To measure both, the paper introduces a belief-rollout diagnostic that elicits structured trajectories across nine belief dimensions, including constraint awareness, failure mode, uncertainty, and repair cost, under alternative harness configurations. Think of it as a flight data recorder that captures not just whether the plane landed but what the autopilot believed about fuel, weather, and runway at every waypoint.

Four specific harness interventions expose the split between outcome and belief: blocked actions, compressed repairs, selective verification, and cost-aware evidence pruning. Each can preserve terminal success while measurably shifting the beliefs that would govern the next task, the next escalation decision, or the next repair attempt. On controlled coding tasks and public-benchmark stress tests, cross-harness belief divergence is not noise. It is systematic, decomposable, and harness-specific.

The paper also ships BIWM, a no-training protocol that addresses this directly. BIWM canonicalizes observations, logs censored branches, expands repair traces, records verification masks, and runs risky branches in shadow before aligning belief trajectories across harness views. It does not require retraining the base model. For teams running agent evaluations, the takeaway is direct: if your harness differs from the benchmark harness, your scores are measuring a different belief regime, not just a different implementation.

We're thinking: We keep seeing agent leaderboards treated as model rankings, but this paper makes the contamination concrete and measurable. The same LLM, solving the same task, through SWE-bench's scaffolding versus your internal harness, is not running the same experiment. It is running two experiments with different observation censorship, different repair compression, and different verification coverage. The belief divergence compounds across steps, meaning the gap between "what the model believes at step 3" and "what it should believe" grows with horizon length. That has direct consequences for any team using benchmark scores to justify production deployment: the score tells you the model can reach the right answer through one specific belief scaffold. It says nothing about whether it will reach the right answer, or take the right intermediate actions, through yours.

Key takeaways:

• Harness design shapes agent beliefs through two separable mechanisms: an immediate interface shift at arrival and a compounding divergence that grows with step horizon, both measurable via structured belief-rollout diagnostics across nine dimensions.
• On controlled coding tasks and public-benchmark stress tests, blocked actions, compressed repairs, selective verification, and cost-aware evidence pruning each preserve terminal success rates while producing measurable cross-harness belief divergence; the gap is systematic, not random noise.
• Teams evaluating agents across frameworks should treat harness configuration as an experimental variable, log censored branches and verification masks explicitly, and consider applying the BIWM alignment protocol before comparing scores across scaffolds.

Source: Measuring Harness-Induced Belief Divergence in Multi-Step LLM Agents