GitHub Secret Scanner vs CLI Tools: gitleaks, TruffleHog, and ShipGuard Compared

14 min · jul 2026

A founder asked us last week whether GitHub's secret scanner had his repo covered. Private repo, free plan. So the answer was: GitHub's secret scanner had never looked at it. Not once. The feature he was trusting runs free on public repositories and costs money on private ones — and nothing in his workflow had ever told him which side of that line he was on.

Answer capsule: GitHub secret scanning is a server-side platform feature — it scans repos hosted on GitHub (full history, all branches, issues, PRs) for 500+ known credential patterns, free on public repos, paid on private ones, and it cannot run on your machine. CLI secret scanners — gitleaks, TruffleHog, ShipGuard — run locally, before a push exists, and each catches a measurably different subset of real leaks. We know, because we planted six secrets in one repo and ran all four. No tool caught everything. The catch/miss table is below, including what our own tool missed.

TL;DR

• We seeded one Next.js + Supabase + Stripe fixture repo with six planted secret conditions and ran gitleaks 8.21.2, TruffleHog 3.83.7, and ShipGuard 0.5.0 for real; GitHub's coverage is documented from its published pattern data, since it can't run locally.
• No tool caught everything. gitleaks caught the widest raw-pattern spread, TruffleHog the fewest here (by design — it optimizes for verified live credentials, not breadth), ShipGuard was the only one to flag the .env file tracked in git as its own finding.
• The legacy Supabase service-role JWT beat every tool: two missed it outright (ShipGuard's miss is a known gap on our fix list), one flagged it identically to a harmless public key, and GitHub documents coverage only for the new key format.
• GitHub secret scanning is free on public repos only; a free-plan private repo — where most vibe-coded apps live — gets nothing without a paid add-on.
• For an AI coding agent's pre-commit loop the deciding factors aren't detection counts: exit codes, JSON output, local execution, zero auth. Only two of the four clear all four bars out of the box.

The four scanners at a glance

Three flags before the experiment. gitleaks' repository now carries a maintenance warning: its author declared it feature-complete in 2026, security patches only, while he focuses on a successor project — it still works, but the rule set has a horizon. TruffleHog's results below look thin partly because of a scope choice we made: verification is its whole thesis, and synthetic keys can't be verified. And ShipGuard is our tool — the fixture, commands, and unedited outputs are published alongside this article so you can re-run everything and check our homework.

What is a secret scanner?

A secret scanner is a tool that reads your code — and, depending on the tool, your git history, CI logs, or S3 buckets — looking for credentials that shouldn't be there: API keys, database passwords, signing keys, provider tokens. When it matches one, it reports the file and line, and the good ones exit non-zero so a pipeline can stop a deploy. Detection comes in three flavors: pattern matching (a Stripe live key always starts sk_live_), entropy analysis (a 64-character random string in a config file is suspicious with no known prefix), and verification (ask the provider's API whether the credential actually works). Every tool here mixes those flavors differently, and the mix — not the marketing — decides what each one catches.

The experiment: one repo, six planted secrets, four scanners

Spec-sheet comparisons of these tools already exist. What we couldn't find anywhere: someone building one repo, planting a known set of secrets, running every scanner, and publishing the catch/miss table — including their own tool's misses. So that's what this is.

The fixture is a minimal Next.js 14 + Supabase + Stripe app: nine files, one git commit, the shape of a thousand AI-scaffolded checkouts. We planted:

1. An AWS access key ID (AKIA + 16 chars) — in .env and duplicated in scripts/deploy.sh, because keys in real repos multiply
2. The paired AWS secret access key (40 chars) — both files, next line
3. A Stripe live secret key (sk_live_51…) — in .env and hardcoded as a fallback in lib/stripe.ts
4. An OpenAI project key (sk-proj-…T3BlbkFJ…) — in .env
5. A Supabase service-role JWT (legacy eyJ… format, "role":"service_role" in the payload) — in .env, with the anon-role JWT beside it as a control, since the anon key is public by design and flagging it is noise
6. A generic high-entropy 64-character token (INTERNAL_DEPLOY_TOKEN) — in .env, no provider format
7. Plus the condition that makes everything above worse: the .env file itself is committed to git.

Every key is format-realistic but synthetic — randomly generated, never issued by any provider, verified dead. That realism matters more than it sounds: our first fixture draft used keys with giveaway words like "FAKE" in them, and TruffleHog's false-positive wordlists correctly filtered them out — which would have unfairly understated its detection. We rebuilt the fixture with realistic values so every tool saw what an actual leak looks like. Fairness cuts in every direction or the numbers mean nothing.

One scanner never touched the repo: GitHub's. It can't — it runs only on GitHub's infrastructure, against pushed code. So we score it from its own published pattern data rather than pretending absence of a local run is absence of capability.

The results: who caught what

Timings on the same machine, warm runs: gitleaks ~80ms, ShipGuard ~1.2s, TruffleHog ~3.5s with verification disabled (its engine reports the scan itself at 53ms — startup dominates). At nine files, speed differences are about developer patience in a pre-commit hook, not capability.

Now the part a spec sheet can't tell you — why each row landed the way it did.

gitleaks caught the widest raw spread: five distinct rule hits, including the one nothing else touched locally — the anonymous 64-character token, via its generic-api-key entropy-plus-keyword rule. Two rows need honesty, though. Its OpenAI hit came from that same generic rule, not the OpenAI-specific one — the dedicated rule predates the sk-proj- key format, so a spec-sheet reading of "has an OpenAI rule" would mislead you. And its jwt rule flagged both Supabase JWTs identically: the service-role key that bypasses your database's row-level security, and the anon key that ships in every page load by design. A match without a meaning — the developer triaging it has to already know the difference, which is exactly the knowledge gap that put the key in the repo.

TruffleHog looks like it lost this table, and that reading would be wrong. It caught exactly what its typed detectors cover — AWS, Stripe, OpenAI, cleanly, in both files — and nothing where no detector fired: no generic JWT detector, no default entropy net. That's the architecture: TruffleHog's bet is that an unverified pattern match is a cost, not a catch, and its 800+ detectors exist to attempt authentication against real provider APIs and tell you this key works right now. Our synthetic keys made that flagship feature untestable by design — testing it would require planting real credentials, which we won't do. So read its column as "what TruffleHog sees before verification," and weigh verification separately: on a legacy repo with hundreds of stale matches, "which of these still authenticate" decides your weekend.

ShipGuard caught the provider keys in both .env and source files — including the AWS secret access key as its own distinct finding in each file, the row gitleaks skipped — and was the only tool to flag .env tracked in git as a finding in its own right, the root condition everything else grows from. It also flagged a missing middleware.ts: an auth check outside this article's scope, because ShipGuard is a release gate that includes a secret scanner, not a secret scanner alone. Two honest misses. The legacy Supabase service-role JWT: no rule for it in the free tier today — a named item on our fix list, and this fixture is now its regression test. The generic token in .env: the entropy check covers source files (a probe confirmed the same token is flagged when moved into config.ts), but .env contents get file-level treatment via exposed_env, not line-level entropy analysis. The fixture's verdict was DO_NOT_SHIP either way; the unedited JSON is in the published captures.

GitHub secret scanning, scored from its own published pattern data: all three provider keys are covered patterns with push protection and validity checks — on a repo it's actually watching. The structural gaps are what the table can't convey. Nothing in your working tree is scanned until it's pushed, so the local-loop use case doesn't exist. And the coverage boundary is a billing boundary: free on public repos, paid Secret Protection on private ones — the free-plan private repo, where most solo-built apps actually live, is the uncovered middle. On the Supabase row, GitHub documents the current sb_secret_… format; the legacy JWT-shaped service keys that pre-2025 projects still carry aren't a listed pattern.

The row that should worry you

Six planted conditions, four scanners — and the most dangerous single credential in the fixture, the service-role JWT that turns Supabase's row-level security into a suggestion, produced two outright misses, one indistinguishable-from-noise match, and one format-boundary asterisk.

That's no fixture coincidence. Pattern scanners key on shapes, and this credential's shape says nothing: the eyJ prefix is shared by every JWT on earth, including ones designed to be public, and the difference between "public by design" and "database admin" is three base64-encoded words deep in the payload. Four independent tools, one shared blind spot, because they all key on the same thing.

The general lesson is the one we keep re-learning in what a real scan finds in vibe-coded apps: a scanner's marketing enumerates what it catches. Its value is decided by what it misses, and whether anyone told you.

The agent-loop dimension: which scanner can your coding agent actually run?

Here's the comparison axis none of the existing gitleaks-vs-TruffleHog pieces cover, and it's becoming the one that matters most: can an AI coding agent run this tool inside its own loop, unattended, and act on the result?

An agent shipping code needs a gate it can invoke and branch on. That imposes four hard requirements no human-oriented review of these tools scores:

• Local execution, no auth. The agent works in a directory, often before a remote exists. A scanner requiring a hosted repo or a login is out of the loop by definition.
• Deterministic exit codes. if scan fails, don't commit requires the failure to be an exit code, not a paragraph. gitleaks exits 1 on leaks. ShipGuard's contract is 0 = scan completed (verdict in the JSON recommendation field), 1 = blocked (with --strict), 2 = config error, 3 = runtime error — "found something" vs. "the scanner broke" matters when nobody's watching the terminal. TruffleHog exits 0 on findings unless you pass --fail (then it's 183) — a default that silently turns a CI gate into a CI decoration.
• Machine-readable output. ShipGuard emits one JSON document on stdout with --json; TruffleHog emits NDJSON; gitleaks writes a JSON report to a file. All parseable.
• No network side effects. TruffleHog's verification transmits candidate credentials to provider APIs to test them. Defensible when a human chooses it; a different thing when an agent does it ten times an hour on strings that might be your customers' keys.

GitHub secret scanning scores zero on the first requirement and exits this comparison — architecture, not criticism. Push protection is a genuinely good server-side backstop for the human push. But it can't be the gate inside an agent's write-test-commit loop, because by the time GitHub sees the code, the loop already ended.

Why does this axis matter more every month? Because the agent is increasingly the one holding the credentials.

"Large language models are software, and if you want to deploy them or build applications on top of them, you need to understand the vulnerabilities and what can go wrong. That includes everyday mistakes, like an agent making the wrong tool call, but also worst-case scenarios where an attacker has an incentive to make your agent misbehave, leak data, or steal credentials."

— Matt Fredrikson, CEO, Gray Swan AI; Professor, Carnegie Mellon University · Red-Teaming after Mythos, Latent Space

An agent that can write lib/stripe.ts can write the key into it — a thousand tutorials hardcode the fallback, so the agent does too. The scanner that catches it has to run where the agent runs, at the agent's speed, and answer in the agent's language: an exit code. Everything else is a dashboard someone checks on Thursday.

So which one do you actually use?

Not one of them. Two — layered by where they run — is the honest answer for most teams. The decision logic, rather than a diplomatic "it depends":

If your repo is public on GitHub: you already have GitHub secret scanning with push protection; leave it on. It's your outermost net, it sees your issues and wikis (which no CLI scans), and its partner program notifies providers about leaks you never even triage. Then add one local scanner so secrets die before push instead of after — the layer GitHub structurally cannot provide.

If your repo is private on a free plan: you have no platform layer at all. A local scanner isn't defense in depth for you; it's the only defense.

For git history audits — "did a key ever touch this repo?" — gitleaks is the fast free sweep, with its maintenance-mode status as a known trade-off. TruffleHog is the deeper audit when the follow-up question is "and does it still work?" — the question that decides whether you're rotating one key or doing incident response.

For the pre-deploy gate, especially agent-driven: this is the job ShipGuard was built for — working tree, locally, exit-code contract, one JSON document, plus the auth/payments/deployment checks a secrets-only scanner doesn't consider its job. Per the table above, if legacy Supabase JWTs are your specific exposure, today you'd pair it with gitleaks' generic JWT rule and your own eyes until our fix lands. We'd rather you know that than discover it.

The layering isn't vendor diplomacy — the table above is the argument. Every architecture here missed something the others caught. The .env-in-git condition was invisible to three of four; the high-entropy token was invisible to a different three of four. Single-scanner confidence is the specific error this experiment was built to price.

If your stack includes Supabase, the service-role key that beat these scanners is the same one that makes row-level security the check every AI-built app skips — that teardown covers the database side of the same failure.

Run the gate on your own repo:

npx @agenticcli/shipguard scan

No login, local-only — your code doesn't leave your machine. Wire it into CI with --strict --json so a finding blocks the merge instead of decorating it. Full rule coverage and tiers: ShipGuard.

What this comparison doesn't cover

Boundaries, stated plainly. We tested current-file and single-commit detection — not deep-history performance on 10,000-commit monorepos, where the gitleaks and TruffleHog architectures differentiate further. We couldn't test TruffleHog's verification or GitHub's validity checks against live keys — that requires planting real credentials, and the synthetic keys are why every number here was safe to publish. We didn't test detect-secrets, git-secrets, or the commercial platforms; the four here are the ones a solo builder or small team actually shortlists in 2026. And one repo is one repo: seven planted conditions chosen to be representative of AI-assisted codebases, not a statistical claim about all leaks. The fixture manifest and unedited outputs are published so you can extend the experiment where we stopped.

FAQ