New: Pressure test your SOC 2 report against buyer requirements
Free check
Modern SAST is three jobs, not one: classical scanners for detection, AI triage to clear false positives, and investigative agents for logic bugs. Here's how they fit.
Classical scanners find the patterns. LLMs decide which ones matter. The combination does what neither can do alone.
Most teams run SAST one way: point a scanner at the codebase, get a list of findings, assign them to engineers. The scanner is tuned for recall, so the list is mostly noise. Engineers learn to ignore it. The finding that mattered gets buried alongside the hundred that didn't.
The instinct is to fix this by replacing the scanner with something smarter. Train a model. Fine-tune on your codebase. Let the LLM do the whole job.
Both directions are wrong. Scanners are fast and deterministic. They match patterns across millions of lines in seconds, and they never skip one. Models read context. They can tell you whether a flagged innerHTML assignment is fed by user input or a build-time constant. Neither replaces the other. You want layers.
At Fencer we run three of them. Each one does one job well enough that the next layer can trust its input.
The first layer is classical SAST. Semgrep, CodeQL, Bandit, whatever your stack calls for. These tools match syntactic patterns: innerHTML assignments, unsanitized SQL concatenation, eval() calls, hardcoded secrets. They're fast, reproducible, and cheap. Run them on a codebase and you get every instance of a pattern back in seconds.
What they can't do is reason about whether a pattern actually matters. A scanner that flags innerHTML doesn't know if the assigned value comes from user input or from a build-time constant. One that flags a SQL string doesn't know if it passes through a parameterized query layer first. That distinction separates a vulnerability from noise, and the scanner can't make it.
High recall, low precision. That's the design. The scanner's job is to find every potentially dangerous pattern. Sorting is downstream. On a real codebase, a scanner run produces hundreds or thousands of findings. Single-digit percentages turn out to be real vulnerabilities.
Think of the scanner as the net. It catches everything. The next two layers decide what to keep.
Each finding from Layer 1 gets a context package: the flagged line, its callers and callees, taint sources, and any relevant configuration. The same evidence a human reviewer would pull up before deciding whether a finding is real.
An LLM reads that context and returns a verdict: true positive, false positive, or uncertain. We follow the LLM4SA approach, classifying each finding three times and taking a majority vote. The model is told to be conservative: when the context is ambiguous, flag or escalate rather than dismiss.
This layer removes noise. It doesn't find new things. It doesn't scan the codebase or look for patterns the scanner missed. It takes the scanner's output and decides which findings deserve human attention. On our benchmark of 142 real findings, the best-performing models cleared 85-93% of false positives while catching every confirmed vulnerability.
We benchmarked 15 models on this triage task. The highest accuracy came from Kimi K3, an open-weights model that cleared more false positives than any Claude or GPT model we tested. But the more interesting result was on the other end: the most capable closed model refused to analyze 8% of the findings outright. Content filters flagged ordinary injection sinks and command-execution paths as dual-use security content.
We're not the only ones hitting this. When Hugging Face responded to the OpenAI model escape incident in July 2026, commercial APIs blocked their analysis requests containing exploit payloads. They ran forensics on an open-weights model locally.
Defensive security work looks identical to offensive security work through a content filter. For triage, where every finding contains code that a scanner already flagged as potentially dangerous, content filtering isn't an edge case. It breaks the pipeline.
Open-weights models running on your own infrastructure give you three things closed APIs can't: no content filtering on your own security findings, predictable costs at scale, and zero dependency on an external service during an incident. For high-volume triage of scanner output, that's the right default.
Not accuracy. Accuracy treats a dismissed vulnerability and a cleared false positive as equivalent. They're not even close.
A good triage model clears false positives aggressively. If it hedges on everything and sends it all to a human, you haven't removed any work. But it never silently dismisses a real vulnerability. One missed vulnerability costs more than a hundred extra false positives on a reviewer's plate. When the model isn't sure, it escalates.
The output matters too. A bare true-positive/false-positive label isn't enough. Engineers reviewing the triage need to see which data flows the model traced, what sanitization it found or didn't find, and why it concluded the finding is or isn't reachable. Trust requires showing the work.
In our benchmark, the models with the highest accuracy weren't always the ones we'd ship. The model we run in production catches every real vulnerability, escalates when uncertain, and clears enough noise that the queue is actually usable.
The first two layers handle patterns the scanner already knows about. Layer 3 goes after a different class of problem.
Logic bugs. Broken authorization checks. Race conditions. IDOR vulnerabilities. Insecure state machines. No scanner rule will find an endpoint that returns another user's data because the authorization check queries the wrong table. No regex will catch a payment flow that allows negative quantities. These live in the application's logic, and finding them requires understanding what the code is supposed to do, not just what it does.
More capable models earn their cost here. The agents get broader context: API route definitions, middleware chains, database schemas, authentication flows. They answer open-ended questions. Does this endpoint enforce authorization correctly? Can this state transition be triggered out of order? Is this business rule actually enforced on every path, or just the one the developer tested?
Layer 3 is slow, expensive, and more speculative. A triage model processes a finding in under a second. An investigative agent might spend minutes reasoning about an authorization flow across multiple files. The cost per investigation runs 10-50x higher and the false positive rate goes up. But the findings it surfaces are the ones nothing else in your pipeline would catch.
It's not a replacement for penetration testing. Human pentesters bring adversarial creativity, business context, and the ability to chain findings across system boundaries in ways current models can't match. Layer 3 catches bugs that sit between what a scanner can pattern-match and what a pentester would find with unlimited time. It narrows that gap, but the gap is still there.
It's also not a first pass. Running an investigative agent across every file in a codebase burns money for little return. It works because the earlier layers have already surfaced the areas where the application handles sensitive operations. The expensive reasoning gets focused on code that earned the attention.
Scanners are fast, cheap, and exhaustive, but they can't reason about context. They'll hand you hundreds of findings, most of them noise, and they'll never find a logic bug.
Triage models can reason about context, but they need a finding to reason about. They can't scan a codebase on their own. They answer "is this specific pattern exploitable?" Not "what else could go wrong?"
Investigative agents can find higher-order vulnerabilities, but they're expensive and slow. You can't run them across an entire codebase. They need the earlier layers to narrow the scope to code that handles sensitive operations.
| Scanners | Triage models | Investigative agents | |
|---|---|---|---|
| Finds | Syntactic patterns | Which patterns are real | Logic and authorization bugs |
| Speed | Seconds | Seconds per finding | Minutes per investigation |
| Cost | Low (fixed tooling) | Low (open-weights, ~$0.005/finding) | High (~$0.05-0.10/investigation) |
| False positive rate | Very high (90%+) | Low (model-dependent) | Moderate |
| Misses | Context, logic, authorization | Novel vulnerability classes | Scale, adversarial chaining |
| Needs | Codebase access | Scanner output + code context | Narrowed scope from earlier layers |
The pipeline is sequential. Each layer reduces the problem for the next. The scanner finds 500 potentially dangerous patterns. The triage model clears 450 as false positives and flags 50 for review. The investigative agent examines the 10 that touch authentication or payment flows and finds 2 logic bugs no scanner rule would have caught.
Take any layer out and something breaks. No scanner, no findings to triage. No triage, your engineers drown in noise. No investigative layer, the bugs that actually get exploited in production stay hidden until a pentester or an attacker finds them.
A pipeline with LLMs in it can drift. Models hallucinate. They invent sanitization that doesn't exist. They assume authorization checks they can't see. Leave a triage model unchecked and it'll talk itself into dismissing findings a human reviewer would flag. Let an investigative agent run unsupervised and it'll surface plausible-sounding logic bugs that aren't real.
Configurable guardrails at every layer keep this in check.
At the triage layer, guardrails define what the model can and can't dismiss on its own. A finding in an authentication flow should need higher confidence to clear than one in a logging utility. A finding where user input flows into innerHTML should never be auto-dismissed, regardless of what the model thinks the server is escaping. These aren't baked into the model. They're configuration your security team owns, tunable per-repository, per-severity, and per-finding-category.
At the investigative layer, guardrails constrain what the agent can access and how far it goes. An agent examining authorization logic needs route definitions, middleware, and database schemas. It doesn't need deploy credentials or production secrets. Scope the context to what's needed. Lock actions to read-only. An investigative agent that can modify code or reach external systems isn't a security tool. It's a liability.
Guardrails prevent bad outcomes. They don't drive good ones. For that, you need ground truth.
The best way to focus an investigative agent is to hand it a threat model. Not a generic OWASP top-10 checklist. Your application's threat model: the assets worth protecting, the trust boundaries, the actors, the attack scenarios your team has already thought through.
That context transforms the investigation. "This application processes payment card data. The trust boundary sits between the API gateway and the payment service. The primary threat is authenticated users trying to access other users' payment methods." Now the agent isn't asking "what could go wrong here?" It's asking "can an authenticated user reach another user's payment data through this endpoint?" Same code, sharper question, better results.
Without a threat model, an investigative agent wanders. It might find interesting things. It'll also burn cycles on low-value paths and produce findings that are technically correct but irrelevant to your actual risk profile. With one, the agent gets the same focus a pentester would have after being briefed on scope.
The threat model pays off across every layer. It identifies assets and trust boundaries. Scanner rules get tuned to flag patterns that cross those boundaries. The triage model knows which findings touch sensitive flows and raises its threshold before dismissing them. The investigative agent targets the attack scenarios the threat model describes. One source of truth, three layers that benefit from it. The output reflects what your team actually cares about, not what a generic scanner rule happens to match.
We're building this at Fencer. The scanner runs on every commit. The triage model clears the noise. The investigative agents focus on authorization and business logic where it counts. If you're working on something similar, we'd like to hear how you're approaching it.
