Coding Agents Beat SOTA on Only 17.8% of Real Science Tasks

Every "AI scientist" demo shows a model reading a paper, writing code, and producing results. The assumption baked into those demos is that running the experiment and advancing the experiment are roughly the same problem. They are not. NatureBench puts a hard number on the gap.

NatureBench draws 90 tasks directly from peer-reviewed Nature-family publications, spanning disciplines from genomics to materials science. Each task is packaged inside NatureGym, an automated pipeline that builds a containerized, reproducible environment from the source paper itself. That containerization step matters: prior agent-on-research benchmarks suffered from environment fragmentation, where different groups ran agents against slightly different setups, making cross-paper comparisons meaningless. NatureGym eliminates that variable. Ten frontier agent configurations ran under a strict web-search-disabled protocol, so models could not retrieve the published answer and reformat it as their own output.

The diagnosis from that setup is structural. Agents do not fail because they misread the task. Failure analysis shows that wrong method choice and insufficient compute budget dominate the error distribution, not task misunderstanding. What agents actually do when they succeed is methodological translation: they reframe a scientific problem as a supervised prediction task, then apply standard ML tooling. That works often enough to clear the bar on familiar problem shapes. It stops working the moment a task requires genuine methodological invention, choosing an approach that does not already exist in the model's training distribution of code patterns.

The headline number is 17.8%. Under the g>0.1 criterion (a meaningful improvement over published SOTA, not just matching it), the strongest evaluated configuration clears the bar on fewer than one in five tasks. For teams building or evaluating AI research assistants, the takeaway is direct: reproduction capability and discovery capability are distinct skills, and current frontier models have the first but not the second.

We're thinking: The 17.8% figure gives the "AI scientist" narrative a specific, defensible ceiling to argue against, and right now that ceiling is low. We find the method-pathway analysis more telling than the headline number: agents succeed by converting hard science problems into familiar supervised learning problems. That is a known strength of current coding agents, not a scientific reasoning capability. It means the benchmark is not just measuring compute or context length. It is measuring whether agents can operate outside the distribution of ML tasks they were trained on. Until that changes, any product claim that an AI agent can "accelerate discovery" deserves a direct question: discovery of what kind, and how often does it actually beat a domain expert's published baseline?

Key takeaways:

• Agents succeed through methodological translation (reframing science tasks as supervised prediction) rather than inventing new approaches; failures trace to wrong method selection and compute budget, not task comprehension.
• The best of ten frontier configurations surpasses published SOTA on 17.8% of 90 Nature-family tasks under the g>0.1 criterion; the containerized NatureGym pipeline controls for environment fragmentation, making this the most credible cross-agent comparison on real research tasks to date, though 90 tasks across all disciplines is a narrow sample.
• Teams evaluating AI research tools should benchmark against published SOTA on held-out domain tasks, not against reproduction accuracy alone, since the two metrics diverge sharply once tasks require methodological novelty.

Source: NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?