skip to content

~/blog/regex-alone-false-positives

Why regex alone drowns you in false positives

Entropy gates, placeholders, fixtures, docs examples, and paired credentials: where pattern matching stops, why context reasoning has to take over, and why a dismissed finding must stay visible.

Devin OkaforCo-founder and CTO5 min read
  • #detection
  • #entropy
  • #false-positives

I spent years building detection pipelines for a code host. The hardest lesson was that recall is cheap. Any intern can write a regex that finds every AWS key in a repo. The expensive part is the other 95 matches (an example figure, but the shape is real) that are not keys, and what they do to the person reading the report.

A scanner that fires 60 times on a repo with 2 real leaks does not have a precision problem. It has an ignored-report problem. The developer skims, sees fixture passwords and docs examples, and closes the tab with the two live keys unread. False positives are how real findings get missed.

What a regex can know

A pattern knows shape. AKIA followed by 16 uppercase alphanumerics is shaped like an AWS access key ID. ghp_ plus 36 characters is shaped like a GitHub token.

But shape says nothing about whether the value is live, whose it is, or what it opens. And half the secrets in real code have no shape at all: a database password, a signing secret, an internal API token. For those, the only pattern is something_secretish = "some string", which also matches every form label, test, and tutorial ever written.

Entropy helps, then stops helping

The standard fix is a Shannon entropy gate. Measure the bits per character of the candidate. A random base64 token of 40 characters lands somewhere around 4.5 to 5.5. An English phrase like changeme-please lands closer to 3. Set a threshold per character set and drop what falls below.

text
candidate                     entropy   verdict by entropy alone
your-api-key-here             3.3       dropped (correct)
AKIAIOSFODNN7EXAMPLE          3.7       kept (wrong, docs example)
c8f1••••••••28e4 (git sha)    3.9       kept (wrong, commit hash)
hunter2                       2.8       dropped (wrong, real password)
wJ7r••••••••5yWn              5.3       kept (correct)

Those figures are approximate, and that is the point. Entropy separates random from non-random. It does not separate secret from non-secret. Commit hashes, UUIDs, integrity checksums in lockfiles, minified bundle chunks, and base64 images are all high entropy and all harmless. A weak human-chosen password is low entropy and very much a finding. We use entropy as a gate on generic candidates and never let it have the last word.

The four usual suspects

  • Placeholders. YOUR_API_KEY, xxxxxxxx, <token>, changeme. Easy to list, endless in variety. Someone always invents a new one.
  • Fixtures. A test file asserting that the parser handles a key-shaped string will contain a key-shaped string. The path says __tests__ or fixtures, and the value often repeats across cases.
  • Docs examples. AKIAIOSFODNN7EXAMPLE is published in the AWS documentation and appears in thousands of repos. It passes the shape check and a naive entropy check. It ends in EXAMPLE, which a regex was never asked about.
  • Format lookalikes. Hashes, UUIDs, object IDs, and encoded blobs that happen to fit a generic length and alphabet.

You can write a rule for each of these. We did, and those rules are the floor Leaksweep falls back to when no model is reachable. But each rule is a small bet about how people write code, and people keep writing code in new ways.

Paired credentials change the math

An AWS access key ID on its own is an identifier. It is roughly as sensitive as a username. The same ID sitting three lines above a 40 character value assigned to aws_secret_access_key is a complete credential, and the severity is not the sum of two mediums. It is critical.

A per-line regex can not see that. It reports two findings with two independent ratings. The same goes for a client ID next to a client secret. Risk lives in the relationship between values, and relationships are context.

What context reasoning adds

This is the part that needs a language model, and the reason we call Leaksweep AI-native instead of AI-assisted. After the pattern pass, each candidate goes to a reasoning layer with a masked value, the file path, and a redacted excerpt of the lines around it. The model never sees the secret. It does not need to. It needs what a senior reviewer would look at:

  • Where is this file? src/config/prod.ts and docs/quickstart.md are different worlds.
  • What is the variable called, and what is done with it? Passed to an SDK constructor, or compared in an assertion?
  • What sits next to it? A matching key ID, a production hostname, a comment saying TODO move to vault?
  • Does the masked head and tail match a known example or a test-mode prefix?
  • Is there an env read with this literal as the fallback? That is a real key wearing a disguise.

The output is a verdict, a severity, and a sentence of reasoning, parsed against a strict schema. We treat it as untrusted input like anything else that crosses a boundary. And we grade it: every prompt or routing change runs against a labeled corpus, and a change that loses a real secret does not ship, no matter how much noise it cuts.

Dismissed findings stay on the page

Here is the rule we will not bend. A finding judged to be noise is still listed, with the reason, every time. It is never hidden, collapsed out of the export, or dropped from the count.

  1. Models are wrong sometimes. If the reasoning says docs example and you know that file is a live deploy script, you need to see the line to overrule it.
  2. Code can lie to the reviewer. A comment reading test key, safe to ignore beside a production credential is cheap to write, by accident or on purpose. A visible dismissal with a stated reason can be audited. A silent one can not.
  3. Trust comes from checking. The first few times you use a scanner, you should be able to verify its cuts. Skimming later is earned, not assumed.

Regex finds candidates. Entropy thins them. Context decides. And the human keeps the right to disagree, which only works if nothing is swept out of sight.

$ leaksweep init

Sweep your code before someone else does.

Three sweeps a month on the free tier. No card. Secrets masked from the first byte.