Cognition's FrontierCode 2.0 Benchmark Shows Every AI Coding Model Fails Over Half of Real PRs

Cognition, the engineering lab behind the autonomous software agent Devin, has released FrontierCode 2.0, an evaluation suite designed to measure whether frontier models can produce genuinely mergeable pull requests from informal, underspecified developer prompts. The results expose a wide chasm between synthetic coding leaderboards and daily production engineering: Anthropic’s Claude Opus 5.5 leads the benchmark at 45%, meaning every AI coding model currently available fails to capture more than half of the available score on realistic repository tasks.

As top models saturated legacy benchmarks like SWE-bench Verified and climbed steeply on DeepSWE v1.1, enterprise engineering leaders noticed a disconnect. A model that scores above 75% when given a detailed GitHub issue description and an isolated unit test harness frequently generates unmergeable pull requests when handed the kind of terse, two-sentence Slack message or Jira ticket that human senior engineers routinely resolve.

FrontierCode 2.0 specifically targets that ambiguity gap. Rather than spelling out every file path, edge case, and cleanup requirement in the prompt, the benchmark gives agents realistic, high-level feature or bug-fix requests inside complex repositories that are 25% larger than those in FrontierCode 1.1. Reference solutions require modifying 67% more files per task, and strict grading rubrics penalize models that leave behind dead code, break public API contracts, or fail to support sibling input types.

The leaderboard also reshuffles model tier rankings and challenges the assumption that burning more test-time reasoning tokens always yields better code. While Claude Opus 5.5 claims the top overall accuracy spot at 45%, OpenAI’s GPT-6.1 Sol takes the cost-efficiency crown by scoring 39% at an average inference cost of just $1.03 per task—and Cognition’s data shows that cranking reasoning effort to maximum frequently lowers mergeability scores due to over-engineering and unrequested breaking changes.

Fast Facts: Cognition FrontierCode 2.0 Coding Benchmark

• Primary Sources & Release: Published by Cognition Engineering in Cognition’s FrontierCode 2.0 release analysis with independent verification in AlphaSignal’s benchmark breakdown.
• Overall Leaderboard Ceiling: Claude Opus 5.5 ranks #1 overall at 45%; no commercial or open-weight model achieves 50% on the mergeable PR rubric.
• Cost-Efficiency Leader: OpenAI’s GPT-6.1 Sol achieves 39% accuracy at an average cost of $1.03 per task, offering the highest score-per-dollar ratio among frontier models.
• Benchmark Contamination vs. Real-World Generalization: Models that posted top-tier scores on DeepSWE v1.1—specifically Muse Wonder 1.0 and Gemini 4 Argon—drop into the middle of the pack on FrontierCode 2.0’s underspecified prompts.
• Open-Source Geo Split: Leading Chinese open-weights models average 17% on FrontierCode 2.0, outperforming US open-source entries such as Beam and Inkling, which score 3%.
• Complexity Jump Over v1.1: Compared to FrontierCode 1.1, repository codebases are 25% larger, ground-truth reference patches touch 67% more files, and “thoroughness blockers” (missing edge-case handling or incomplete multi-file updates) tripled (3x).

Leaderboard Comparison: FrontierCode 2.0 Accuracy, Cost, and Failure Modes

The table below summarizes model performance, task economics, and behavioral patterns across key commercial and open-weight tiers evaluated on FrontierCode 2.0.

Why Models Fail Half of Real PRs: The Three Thoroughness Blockers and the Reasoning Paradox

Cognition’s error analysis explains why autonomous coding agents often frustrate senior human reviewers even when the generated code compiles and passes basic unit tests. In FrontierCode 2.0, evaluators tripled (3x) the density of “thoroughness blockers”—structural requirements that are never explicitly written in a user’s prompt because any competent human maintainer would infer them from the surrounding codebase.

Across the evaluated trajectories, models lost more than half of available points to three recurring engineering failures:

1. Failure to Reuse Existing Repository Abstractions: When asked to add a new webhook handler, serialization format, or database query, models frequently write bespoke helper functions from scratch inside the target file rather than discovering and importing the canonical utility class already used across the other 25%-larger repository modules.
2. Ignoring Sibling Input Types and Call Sites: With reference patches modifying 67% more files than in FrontierCode 1.1, a valid fix rarely lives in a single function. If a prompt asks to “fix timestamp parsing in the export pipeline,” a mergeable PR must update CSV, Parquet, and JSON exporters alike, preserve public API signatures, and maintain resource cleanup guarantees. Models routinely patch the primary path mentioned in the prompt while leaving sibling code paths broken.
3. Omitting Regression Tests: Unless explicitly instructed to write tests in the prompt, many models submit implementation diffs without adding corresponding coverage to the repository’s existing test suite.

Perhaps the most counterintuitive finding in the FrontierCode 2.0 report is the Reasoning Effort Paradox. Engineering teams often assume that setting a reasoning model’s thinking budget to high or max will improve pull request quality. Cognition found the opposite on underspecified tasks: cranking up reasoning effort frequently lowered scores. When given excessive chain-of-thought budgets on a short, informal prompt, models begin inventing unrequested product requirements, refactoring unrelated modules, and introducing breaking API changes that cause human maintainers to reject the PR outright.

The benchmark also exposes overfitting on older static suites. Models such as Muse Wonder 1.0 and Gemini 4 Argon, which climbed near the top of DeepSWE v1.1 leaderboards, regressed to the middle of the pack on FrontierCode 2.0—demonstrating that optimizing reinforcement learning post-training for highly detailed benchmark prompts does not transfer automatically to ambiguous real-world tickets. Meanwhile, in the open-weight ecosystem, Chinese open models (17%) have opened a nearly 6x capability lead over US open-source contenders like Beam and Inkling (3%).

Engineering Workflows to Overcome the 45% Autonomous PR Ceiling

Because even Claude Opus 5.5 (45%) and GPT-6.1 Sol (39%) fail more than half of end-to-end PR criteria when handed raw user tickets, engineering organizations must redesign how prompts are expanded and validated before an agent writes code.

The 4-Step Enterprise Implementation Playbook

1. Insert an Automated “Prompt Specification Expander” Before Code Generation: Never pass a raw one-line Jira ticket or Slack message directly to a coding agent. Build a lightweight pre-planning stage that scans the repository index to append explicit pointers to existing helper abstractions, sibling classes that share the same interface, public API stability constraints, and the exact test file where regression tests must be added.
2. Route Routine PRs to GPT-6.1 Sol ($1.03/Task) and Reserve Opus 5.5 for Deep Refactors: At 39% accuracy for $1.03 per task, GPT-6.1 Sol delivers 86% of Claude Opus 5.5’s top-ranked capability (45%) at a fraction of flagship token cost. Use GPT-6.1 Sol as the default worker for daily ticket triage and bug fixes, escalating to Claude Opus 5.5 only when tasks span complex multi-module architecture boundaries.
3. Cap Reasoning Effort on Underspecified Feature Requests: To prevent the over-engineering and scope creep documented in FrontierCode 2.0, configure agent orchestrators to use medium reasoning budgets during implementation phases. If higher reasoning effort is needed to diagnose a root cause, restrict the high-effort call to a read-only diagnostic pass, then lock the coding pass to a strict file-scope whitelist.
4. Enforce Deterministic Diff Linters for Thoroughness Blockers: Before an agent opens a GitHub pull request for human review, run an automated CI gate that checks for the top FrontierCode 2.0 failure modes: reject any agent PR that modifies implementation files without touching a test_* file, alters exported function signatures without an explicit breaking-change label, or duplicates existing utility functions detected via AST similarity.