New: Pressure test your SOC 2 report against buyer requirements
Free check
We ran Claude Code's /security-review over 50 real pull requests. It is a useful first pass for catching vulnerabilities, but not a full security program. Here's why.
AI coding agents can now review their own output for security issues. But do the AI coding agents' built-in security reviews give you the security infrastructure you need?
Claude Code ships /security-review. Codex runs security checks on generated code. Other AI coding agents are building similar capabilities. The pitch is compelling: the same model that wrote the code reviews it for vulnerabilities, with full session context, instant feedback, and zero tool switching. You don't leave the terminal. You don't configure a scanner. You just ask the agent to check its own work.
We've written before about whether agents leak secrets they encounter during coding tasks. That's agent behavior security: does the agent itself become the threat? This is a different question. When an agent writes code and then reviews that code for vulnerabilities, can you trust the review?
To find out, we ran /security-review on 50 pull requests across three production repositories over four weeks. Not synthetic examples. Real PRs with authentication flows, API endpoints, database queries, file uploads, and infrastructure changes. We ran each review multiple times and tracked what it found, what it missed, and what changed between runs.
The results were mixed in a way that matters.
Agent-native review catches real issues. On the 50 PRs we tested, it flagged:
These are the patterns that experienced reviewers catch on a first pass, and the agent catches them too.
The reviews give useful explanations for the issues they find. When it flags an issue, it explains the data flow, identifies the sink, and suggests a fix. For a developer who isn't a security specialist, this is more educational than a scanner that says "CWE-79: Cross-Site Scripting on line 42." The agent tells you why line 42 is dangerous and what to do about it.
For individual developers working on side projects, or teams in early stages without dedicated security tooling, this is an improvement. However, it's important to be aware of the limitations of these agent reviews. Knowing what is not covered is perhaps more important than knowing what is.
We ran /security-review on the same PR three times in separate sessions. The first run flagged four issues. The second flagged three, two of which overlapped with the first run. The third flagged five, including two that neither previous run caught.
This happened consistently. Across 50 PRs with repeated reviews, findings appeared and disappeared between runs. A missing rate-limit check showed up in 2 of 3 runs. A path traversal risk appeared once in three attempts. An IDOR vulnerability was flagged consistently, then missed entirely on a fourth run we did as a sanity check.
For a security tool, non-determinism is disqualifying at the foundation layer. Determinism is what lets you reason about coverage. "We scanned this codebase and found zero critical issues" means something when the scanner produces the same output every time. It means nothing when the same code might produce a different finding set tomorrow.
A vulnerability that appears in 2 of 3 reviews isn't a flaky test you can retry. It's uncontrolled coverage. You don't know what you're missing because the tool doesn't know either.
Every /security-review invocation is a frontier model call. The cost scales with the size of the diff, the length of the conversation context, and the model's token pricing, none of which correlate with the actual security risk of the change.
A 2,000-line refactor that renames variables and reorganizes imports costs the same to review as a 50-line change to your authentication middleware. The refactor has near-zero security surface while the auth change is quite significant. The LLM processes both with the same depth, burning the same tokens.
At 20 PRs per day across a small engineering team, the monthly cost of LLM-based security review on every PR exceeds what many dedicated SAST platforms charge annually. And the cost is unpredictable: a week of large refactors spikes the bill with no corresponding increase in security coverage. Dedicated scanners have fixed per-scan costs. You can budget for them. You can't budget for "however many tokens the model decides to use this month."
A full-context /security-review takes 30 seconds to over two minutes depending on the diff size and session history. A deterministic SAST scanner returns findings in seconds, often sub-second for incremental scans on changed files.
In CI/CD, this difference changes developer behavior. A check that adds 90 seconds to every PR becomes the thing developers skip on "low-risk" changes. But "low-risk" is a judgment call made before the review, which is exactly when you want the review to override your assumptions. A security check that developers learn to bypass on routine changes is worse than one that runs silently on every commit, because it creates a false sense of coverage. "We have security review" becomes true for the PRs you remembered to run it on.
Two developers running /security-review on the same file in separate sessions may get different results. Session context, conversation history, and even the order in which files were discussed can influence what the model flags. When the underlying model updates (which happens without notice), the same code that passed review last week might fail this week, or vice versa.
This makes it impossible to establish a security baseline. "We reviewed all code with /security-review and it passed" means something different when the tool's behavior is a function of session state and model version. We measured this systematically when benchmarking 15 models on 142 real findings: models disagree with themselves across runs, and model updates shift the boundary between "flagged" and "cleared" without warning. The same structural problem applies when the model is reviewing code rather than triaging scanner output.
When /security-review flags a potential vulnerability, that finding exists in the chat session and nowhere else. Close the terminal, and it's gone. There is no dashboard showing open findings across repositories, no lifecycle tracking for security findings, and no way to see whether your security posture is improving or degrading.
When a SOC 2 auditor asks "show me evidence that your team reviews code for security issues and tracks findings to resolution," a chat transcript is not an answer. The auditor wants to see a finding record: who discovered it, when, what was the disposition, who approved the disposition, and what evidence supports the decision. That requires infrastructure that agent-native review doesn't provide and isn't designed to provide.
The gap between "the agent flagged something in a chat session" and "the organization tracked a vulnerability to resolution with auditable evidence" is filled with spreadsheets, Jira tickets manually created from chat output, and Confluence pages assembled weeks later when someone asks for proof. This is the same gap we identified across every SAST vendor we compared: finding vulnerabilities is a partially solved problem. Tracking them to resolution with evidence is where the industry still drops the ball.
Even if every gap above were solved tomorrow, LLM-based review would still be one technique. A real security program layers multiple approaches:
Agent review is strong at reasoning about intent and context. "This looks like it's supposed to be an admin-only endpoint, but nothing enforces that." Scanners are strong at exhaustive coverage. "Every instance of innerHTML assignment in the codebase, found in 200ms." These capabilities complement each other. Neither replaces the other.
The danger is when teams treat agent review as sufficient and skip the rest. "We have Claude reviewing our code" becomes the new "we have unit tests" in terms of false confidence. Unit tests verify behavior. They don't verify security. Agent review checks for security issues. It doesn't guarantee coverage.
The six gaps above aren't complaints about implementation quality. They're structural properties of LLM-based review. A dedicated security platform addresses them by separating the things LLMs are good at (contextual judgment) from the things deterministic systems are good at (exhaustive, repeatable coverage).
| Gap | Agent-native review | Dedicated platform |
|---|---|---|
| Determinism | Different findings each run | Same rules, same findings, every time |
| Cost | Scales with LLM token usage per review | Fixed cost per scan, predictable monthly spend |
| Speed | 30 seconds to 2+ minutes per review | Sub-second for incremental scans |
| Repeatability | Varies by session, model version, context | Versioned rulesets, pinned scanner versions |
| Tracking | Findings live in chat session | Persistent vulnerability records with lifecycle |
| Coverage | Single technique (LLM reasoning) | Multiple scanners: SAST, SCA, secrets, DAST |
At Fencer, we built the platform around the principle that these two layers belong together but should stay separate.
Deterministic scanners find. AI triages. Classical scanners run on every commit and catch every instance of a dangerous pattern, deterministically. The same codebase produces the same findings every time. The AI triage layer then reasons about which findings are real and which are noise, using the same LLM judgment that makes agent review useful, but applied downstream of exhaustive detection rather than as the detection mechanism itself. We published benchmarks on this pipeline: 15 models, 142 findings, with public data on which models dismissed real vulnerabilities and which struck the right balance.
Findings persist and have a lifecycle. Every finding becomes a vulnerability record: new, in review, resolved, or risk-accepted. The trend line across repositories shows whether your security posture is improving. When an engineer dismisses a finding, the reasoning is recorded. When someone accepts risk, the decision is tracked with who approved it and when. This is the infrastructure that turns "we found something" into "we tracked it to resolution."
Compliance evidence is a byproduct. Every triage decision, disposition, and review produces auditable evidence automatically. When the SOC 2 auditor asks for proof that your team reviews and dispositions security findings, you export the review log. You don't reconstruct it from chat transcripts and Jira tickets.
We should be honest about the trade-offs in the other direction. Deterministic scanners miss context-dependent bugs that LLM review catches. A scanner won't tell you that an endpoint is supposed to be admin-only based on the surrounding business logic. Our investigative agents attempt this kind of deeper analysis, but they're expensive (10-50x the cost of triage per finding) and more speculative than deterministic scanning. And no combination of scanners and AI replaces a penetration test. The answer is layering, not replacement in either direction.
Agent-native security review is a genuine step forward. It catches real issues, explains them well, and meets developers where they work. Don't remove it from your workflow. It belongs in the inner loop as fast contextual feedback.
It is not a security program. A security program needs determinism, tracking, coverage across multiple techniques, and evidence for auditors. No chat-based review provides these. Treating it as sufficient creates the same false confidence as treating unit tests as sufficient.
Non-determinism is the structural flaw. Cost, speed, and tracking gaps could be engineered around with better tooling and integrations. Non-determinism cannot. It's inherent to how LLMs process prompts. A tool that might catch a vulnerability is a useful extra layer. It is not the foundation you build a security posture on.
Layer, don't replace. Run agent review for fast contextual feedback during development. Run dedicated scanners for exhaustive, deterministic coverage on every commit. Track findings in a platform that produces evidence. Each layer catches what the others miss.
The "one tool" trap is the real risk. Teams adopting AI coding agents are tempted to consolidate. "The agent writes the code and reviews it" sounds efficient. It is efficient for individual productivity. It is not sufficient for organizational security. The gap between "a developer checked" and "the organization tracked a finding to resolution" is where security programs succeed or fail.
If you're evaluating how agent-native review fits alongside dedicated security tooling, the triage benchmark data is public and the SAST comparison walks through the detection, triage, and review layers in detail. We'd rather teams make this decision with data than with vendor promises, including ours.
