skip to content

~/blog/deleted-key-still-in-git-history

You deleted the key. It is still in your git history.

Why removing the line is not remediation, how to find a secret with git log -S, how to purge it with git filter-repo, and why rotation is the only real fix.

Maren HollowayCo-founder and CEO5 min read
  • #git
  • #history
  • #rotation

The most common response to a leaked key is also the least useful. Someone spots the literal in a config file, deletes the line, commits with a message like remove key, and pushes. The diff looks clean. The file looks clean. The problem is fully intact.

Git is an append-only history of snapshots. A new commit does not alter the old ones. The commit that introduced the key still exists, still holds the full value, and is still reachable by anyone who can clone the repo. You have not removed the secret. You have added a signpost that says where to look.

Rotate first. Everything else can wait.

Before any git surgery, revoke the key at the issuer and create a new one. This is the only step that reduces risk. If the repo is public, assume the key was copied within minutes of the push, because scrapers follow public commit feeds in near real time. If the repo is private, assume it has been cloned to laptops you do not control, CI caches you forgot about, and at least one backup.

Find every place it lives

You need to know which commits and which paths carry the value. Git has a search built for this, often called the pickaxe. -S finds commits where the number of occurrences of a string changed, which means the commits that added or removed it.

bash
# every commit on every ref that added or removed the string
git log -p --all -S 'sk_live_' -- .

# same idea with a regex, useful when you only know the shape
git log -p --all -G 'AKIA[0-9A-Z]{16}'

# which branches and tags contain a given bad commit
git branch -a --contains <sha>
git tag --contains <sha>

Search for a distinctive prefix or fragment, not the whole secret, so the full value does not land in your shell history. Note --all. The key may be gone from main and alive on a stale feature branch or behind a release tag. Note the paths too: a key that was committed in .env, then moved into config/prod.json, then deleted, lives at two paths.

Purge with git filter-repo

git filter-repo is the tool the Git project itself recommends in place of filter-branch. It is fast, and its defaults are careful. Start from a fresh clone. It will refuse to run on a repo that does not look fresh unless you force it, and that refusal is a feature.

There are two cases. If the whole file should never have existed in the repo, remove the path from all of history:

bash
git clone git@github.com:acme/api.git api-clean && cd api-clean
git filter-repo --invert-paths --path .env --path config/prod.json

If the secret sat inside a file you need to keep, rewrite the value wherever it appears. Put the replacement rules in a file outside the repo:

bash
# ../expressions.txt, one rule per line:
#   literal value, then ==>, then the replacement
#   THE_FULL_OLD_VALUE==>REMOVED_SECRET
#   regex:AKIA[0-9A-Z]{16}==>REMOVED_AWS_KEY_ID

git filter-repo --replace-text ../expressions.txt

That file holds the full secret in plain text, so it must never sit inside the working tree, and you delete it when you are done. Afterwards, run the git log -S search again. It should return nothing.

Force push, then deal with the people

filter-repo removes the origin remote on purpose, so you can not push a rewrite by reflex. Add it back and overwrite every branch and tag:

bash
git remote add origin git@github.com:acme/api.git
git push origin --force --all
git push origin --force --tags

Every commit after the first rewritten one now has a new SHA. That has consequences your team needs to hear before you push, not after:

  • Collaborators re-clone. Do not pull into an old clone. One git pull followed by a push from a stale clone merges the old history back in, secret included.
  • Open pull requests built on old SHAs need to be rebased onto the new history or recreated.
  • Branch protection may block the force push. Lift it for the push, then put it back the same minute.
  • CI caches and deploy mirrors hold old objects. Clear them.

What you can not rewrite

This is the part that people skip, and it is why purge is not a fix.

  • Forks. Every fork holds its own copy of the old commits. You can not rewrite someone else's fork.
  • Cached views on the host. On GitHub, a commit that no branch points to can still be served by its SHA, and pull request refs keep old commits alive. Removing those takes a request to GitHub Support. On GitLab, run repository cleanup after the rewrite so unreachable objects are pruned.
  • Clones. Any machine that fetched before your push still has the value on disk.
  • Scrapers, search indexes, and archives. Whatever was copied while the commit was public stays copied.

You control exactly one copy of your history. The secret was in all of them.

Why rotation is the only real fix

A secret is a value that grants access because a server agrees it does. Rewriting history changes who can read the value. Rotation changes whether the value works. Only the second is within your power to do completely, in one place, in about a minute, with certainty about the result.

So the order is fixed. Rotate at the issuer. Check the issuer's audit log for use you do not recognize between the leaking commit and the rotation. Ship the new value from the environment, not from a file in the repo. Then purge history so the dead key stops showing up in scans and stops describing your setup to strangers. Then add a pre-commit hook and a CI check, because the next leak will come from the same kind of ordinary afternoon this one did.

When Leaksweep finds a secret, the remediation steps arrive in that order for that reason, with the filter-repo command already filled in for the path it found. The first step always says rotate. It will keep saying rotate even if you tell us the line is already deleted. Especially then.

$ leaksweep init

Sweep your code before someone else does.

Three sweeps a month on the free tier. No card. Secrets masked from the first byte.