Pass/Fail Hides the Real Signal: AgentLens Evaluates the Whole Trajectory

Binary task success is the wrong unit of measurement for coding agents. A run that passes the final test may have taken five wrong turns, ignored tool output, and recovered by luck. A run that fails may have followed instructions precisely, used its tools correctly, and hit a genuinely hard edge case. Both look like a single bit when the benchmark is done with them.

The gap between task outcome and trajectory quality is not a minor calibration issue. It is structural. Agents are interactive systems: they read instructions, call tools, interpret results, self-correct, and communicate with the person waiting on the other end. Collapsing that entire behavioral sequence into a pass/fail score discards most of the signal that matters for production deployment.

AgentLens addresses this directly. Each run is evaluated across the full trajectory, not just its terminal state. Where an objective check exists, formal verification handles it. Where behavior is harder to specify, LLM-written trajectory reviews and side-by-side comparisons fill the gap. The result is not just a score but a readable explanation of why the score is what it is, covering instruction-following, tool use, self-verification, error recovery, and communication quality. Think of it as a code review for the agent's entire session rather than a test suite that only checks the final diff.

The architecture is explicitly production-oriented. AgentLens runs as a nightly evaluation pipeline, designed to catch regressions between successive agent versions before they reach users. That framing matters: this is not a leaderboard tool. It is a diagnostic instrument, and the two goals are different enough that optimizing for one often undermines the other.

No single headline accuracy number defines AgentLens, because the point is that no single number should. The benchmark is open-source at github.com/agent-lens/agent-lens-bench. For teams shipping coding agents into production, the takeaway is direct: if your evaluation pipeline ends at pass/fail, you are flying without instruments on the dimensions that determine whether users trust and keep using the agent.

We're thinking: We find the framing here more consequential than the tooling. The deeper problem AgentLens names is that teams have been optimizing coding agents against signals that do not track user experience. A model that scores well on SWE-bench by brute-forcing test cases may behave erratically, communicate poorly, and fail to recover from tool errors in actual use. AgentLens makes trajectory quality measurable, which means teams can now detect the specific failure mode, not just the failure. The nightly regression pipeline design is worth copying even before the benchmark numbers mature: shipping agent versions without behavioral diffs is the equivalent of deploying code without test coverage, and most teams are doing exactly that.

Key takeaways:

• AgentLens replaces the binary pass/fail signal with full trajectory evaluation, combining formal verification with LLM-written reviews that score instruction-following, tool use, self-correction, and communication across each run.
• The benchmark is open-source and designed for nightly production regression testing across successive agent versions; it does not yet publish a canonical leaderboard with broad model coverage, so external comparability is limited at this stage.
• Teams running coding agents in production should audit whether their current evaluation pipeline captures behavioral regressions between versions, and treat AgentLens as a starting template for building trajectory-level test coverage before the next release cycle.

Source: AgentLens: Production-Assessed Trajectory Reviews for Coding Agent Evaluation