# AWS Says Its Continuum AI Security Agent Fixes 89% of Bugs — Here’s the Catch for DevSecOps

Amazon Web Services (AWS) has made **AWS Continuum**—its autonomous AI security agent for finding, validating, and patching software vulnerabilities—generally available after touting an eye-catching headline metric: an **89.0% end-to-end success rate** (passing 819 of 920 tasks) on the public CyberGym-E2E security benchmark, alongside an offensive penetration-testing run where the agent autonomously exploited **33 of 37 newly disclosed real-world CVEs (89.2%)**.

For engineering and DevSecOps teams drowning in static analysis (SAST) false positives, an agent that automatically scans your repositories, checks findings against your live AWS configuration, reproduces the crash, and opens a verified pull request sounds like an instant fix for security backlogs. AWS CEO Matt Garman highlighted the 89% score—which beat the previous public record of 65.9%—as proof that autonomous agents are shifting the advantage toward defenders.

However, a closer inspection of AWS's benchmark breakdown reveals a crucial catch that every security architect needs to understand before turning an autonomous patching agent loose on production repositories. On CyberGym-E2E, the agent is given 90 minutes to find and repair *any* crash-inducing flaw in 139 open-source C and C++ projects without being told which historical bug the benchmark is grading. While Continuum successfully found, triggered, and patched *a* real bug in 89.0% of runs without breaking existing tests, its patch fixed the **specific historical vulnerability** the benchmark targeted in only **37.8% of tasks**.

Similarly, in AWS's separate 37-CVE offensive penetration-testing trial (powered by Claude Opus 5 with a 10-hour limit per target), the two "misses" occurred because the unguarded benchmark agent pushed its exploit payloads so aggressively that it **bricked the target PHP runtime and knocked the application offline** before it could read the capture-the-flag token.

## Fast Facts: AWS Continuum GA & The CyberGym-E2E / 37-CVE Benchmark Reality

- **General Availability Launch**: AWS Continuum for code vulnerabilities launched in late October 2026, integrating repository scanning, AWS environment validation, and automated pull-request generation, as analyzed by [The New Stack](https://thenewstack.com/aws-says-its-ai-security-agent-fixes-89-of-bugs-but-theres-a-catch/) and the [AWS Security Blog](https://aws.amazon.com/blogs/security/aws-security-agent-now-part-of-aws-continuum-exploited-33-of-37-new-real-world-cves/).
- **The 89.0% CyberGym-E2E Score**: Passed **819 of 920 tasks** within a 90-minute limit (rising to **93.7%** with extended time), beating the prior public high of **65.9%**.
- **The 37.8% Target-Bug Catch**: Across CyberGym-E2E's four grading stages, Continuum reproduced a flaw in **92.5%** of runs, stopped the crash in **89.6%**, kept existing tests passing in **89.0%**, but matched the benchmark's specific historical CVE in only **37.8%** of tasks (vs. the prior **26.2%** record).
- **Language Scope Limit**: Every task in CyberGym-E2E involves **C or C++ memory-safety bugs** across 139 open-source projects—meaning the 89% score does not directly measure business-logic flaws in Python, TypeScript, Java, or Go web apps.
- **33 of 37 Live CVE Exploits (89.2%)**: On 37 containerized real-world CVEs across 8 vulnerability classes, Continuum captured 35 flags and achieved **33 audited end-to-end exploits** in a median time of **~4 hours**.
- **Why 2 Targets Crashed**: In the 2 missed CVEs, running without production non-destructive guardrails caused the agent to brick a PHP runtime layer (`100% non-functional`) while attempting file-upload-to-RCE exploitation.

## Benchmark Stage Breakdown: What AWS Continuum's Numbers Actually Measure

Looking at the stage-by-stage funnel across both AWS Continuum evaluations shows exactly where autonomous security agents excel today—and where human security engineers still have to step in.

| Benchmark & Evaluation Stage | AWS Continuum Score | Prior Public Benchmark High | Practical Meaning for Engineering Teams |
|---|---|---|---|
| **CyberGym Stage 1: Find & Reproduce a Crash** | **92.5%** | ~72% | Excellent at fuzzing and triggering real C/C++ memory faults from scratch |
| **CyberGym Stage 2: Patch Stops the Crash** | **89.6%** | ~68% | Nearly 9 in 10 generated patches successfully eliminate the newly found crash |
| **CyberGym Stage 3: Existing Unit Tests Pass** | **89.0%** (`819/920`) | **65.9%** | Fixes compile cleanly and pass the repository's existing regression suite |
| **CyberGym Stage 4: Matches Target Historical Bug** | **37.8%** | **26.2%** | In blinded discovery, the agent often finds and fixes a *different* bug in the repo |
| **Live CVE Pentest: Audited End-to-End Exploit** | **89.2%** (`33/37`) | N/A (Internal 37-CVE Suite) | Chains 6+ multi-step exploit primitives autonomously in ~4 hours median time |

## Two Practical Lessons for Teams Evaluating Autonomous AppSec Agents

None of these nuances make AWS Continuum any less impressive—beating the previous CyberGym record on both general bug repair (89.0% vs. 65.9%) and historical bug matching (37.8% vs. 26.2%) is a major engineering step forward. However, understanding *how* the benchmark works prevents costly deployment mistakes:

### 1. Blind Bug Hunting vs. Targeted CVE Remediation

In CyberGym-E2E, the agent is dropped into one of 139 C/C++ codebases cold—with no CVE description, no stack trace, and no proof-of-concept exploit. Because mature C/C++ projects often contain multiple latent memory bugs, Continuum frequently discovered, reproduced, and fixed a *different* real vulnerability than the one the benchmark authors had in mind.

For enterprise teams, that means you should evaluate Continuum in two separate modes:

- **Proactive Zero-Day Fuzzing**: Letting the agent hunt blindly for unknown crashes in C/C++ or native modules (where finding *any* exploitable bug is a win).
- **Directed Ticket Remediation**: Feeding the agent a specific CVE advisory, stack trace, or Snyk/Dependabot alert and verifying that its pull request closes that exact vector rather than merely masking a symptom.

### 2. Why Non-Destructive Guardrails Are Mandatory for Pentesting Agents

In AWS's 37-CVE offensive benchmark, the agent ran inside disposable Docker containers under capture-the-flag rules. On one file-upload-to-RCE challenge, the agent pushed payloads hard enough that it permanently wedged the PHP runtime, writing in its own log: *"I've exhausted read/write vectors on the bricked PHP layer... the app is already 100% non-functional."* While AWS enforces non-destructive guardrails in production customer engagements, the incident proves why autonomous pentesting agents must **never** be pointed at live production infrastructure.

## Practical Checklist: Deploying AI Security Agents in Your CI/CD Pipeline

1. **Require Reproducer Tests in Every Agent Pull Request**: Never merge an AI-generated security patch just because existing unit tests pass (Stage 3). Require the agent to commit the failing proof-of-concept test that reproduces the specific CVE (Stage 1) alongside the fix.
2. **Run Autonomous Pentesting Strictly Against Ephemeral Staging Clones**: Even with vendor safety toggles enabled, run active exploitation agents (like AWS Continuum or page-level pentest agents) against isolated container snapshots rather than shared staging or production databases.
3. **Benchmark on Your Actual Language Stack**: Remember that CyberGym-E2E tests C/C++ memory safety; run a 2-week pilot on your own Python, TypeScript, Go, or Java microservices to measure how well the agent handles authentication, IDOR, and business-logic flaws.