# Claude Code or Codex for Security Review? We Tested It

Canonical: https://agenticcli.dev/blog/claude-code-security-review
Published: 2026-11-15

**Short answer:** Claude Code and Codex read your change like a colleague, and both named the open admin route every run. But Claude Code gave 2 different answers in five runs, and 3 runs left a hard-coded live key out. A fixed rulebook reported the same 3 problems 5 of 5 times. Use both: rules first, an AI review second, you last.
A live key sat in the code. We asked Claude Code's security review to look, five times, on the same change. Three times it left the key out of its findings.

The key is a fake value in the real live-key format, planted on purpose in a small Next.js and Supabase shop. Codex, run five times on the same change, left it out twice. A fixed rulebook reported it in all 5 runs.

Nobody was wrong about the verdict: all 10 AI reviews said do not ship. What moved was which problems got named. Ship on the quiet run and you ship the key.

These are our own runs, in our own terminal. The terminal windows below are real output, not mock-ups. Circled numbers mark the lines in question. Spoken quotes come from YouTube's auto-generated captions.

[Video: Can You Rely on Only Claude Code or Codex for Security Review? We Actually Tested It](https://www.youtube.com/watch?v=pHotjTcJUPU)

**How we tested.** Same machine, five runs each, one 5-file change with four planted problems: a hard-coded live key, an admin route with no login, a weak password-reset token and a coupon that can push a total below zero.

| Tool | Version | Model and setting |
|---|---|---|
| ShipGuard CLI | 0.11.0 | fixed rules, offline, no model |
| Claude Code `/security-review` | 2.1.322 | claude-opus-5-5, medium effort |
| Codex CLI | 0.167.1 | gpt-6.1-sol, medium effort, read-only |

In runs 2, 3 and 4 the `/security-review` skill also ran a claude-sonnet-5-5 step inside it; those are the three runs that left the key out. Runs 1 and 5 used Opus only. We cannot say whether that step is the cause. And Codex answered a one-sentence prompt, which limits what it can name. With a findings-list prompt it named the live key and the coupon bug in 5 of 5 runs.

## Why does everyone want an AI to do the reading?

Your agent opens pull requests faster than you can read them. So you hand the reading to an AI. What did you just stop checking?

Matt Pocock named the pressure in his AI Engineer talk on the pull request bottleneck:

> If you just have permanent acceleration pushing stuff through your factory, you're going to end up with a slop cannon, right?

[![Thumbnail of the YouTube video 'Fixing the PR Bottleneck' by Matt Pocock (AI Hero) at AI Engineer; click to watch the moment at 2:21 where he says: 'If you just have permanent acceleration pushing stuff through your factory, you're going to end up with a slop cannon, right?'](/blog/claude-code-security-review/thumb-yt-pocock.webp)](https://www.youtube.com/watch?v=LlgiOCmFG_w&t=140s)

*Matt Pocock (AI Hero), [Fixing the PR Bottleneck](https://www.youtube.com/watch?v=LlgiOCmFG_w&t=140s), AI Engineer, uploaded 2026-09-25, at 2:21.*

Addy Osmani saw what teams did next:

> Reviewers simply could not keep pace with the volume, so code began merging unread, and that became normal.

His one figure comes from Faros AI, a vendor selling into this market, so read it with that in mind:

> PRs merged with zero review up 31.3%

[![Addy Osmani's blog post 'Agentic Code Review', 2026-06-15: 'PRs merged with zero review up 31.3%' and 'Reviewers simply could not keep pace with the volume, so code began merging unread, and that became normal.'](/blog/claude-code-security-review/shot-web-addy-review.webp)](https://addyosmani.com/blog/agentic-code-review/)

*Addy Osmani, [Agentic Code Review](https://addyosmani.com/blog/agentic-code-review/), 2026-06-15.*

Handing the reading to a model is reasonable. [Our agent security guide](/blog/ai-agent-security) covers how much an agent may do alone. Here: what does the model read well?

## What are Claude Code and Codex security reviews great at?

It reads your change the way a colleague would. Anthropic's [command reference](https://code.claude.com/docs/en/commands) describes `/security-review` like this:

> Analyze the changes on your current branch for security vulnerabilities. Reviews the diff between your branch and origin’s default branch, identifying risks like injection, auth issues, and data exposure.

[Codex](/guides/codex) does the same job when you ask in plain words. Both read the diff and ask what the change is for. A rulebook does neither. For what Claude Code writes in the first place, see [our audit of its output](/blog/claude-code-output-security-audit).

### Does it understand what a change is for?

One line opened an admin route that exports every customer. Ten of ten AI reviews caught it.

The route, `app/api/admin/users/route.ts`, returns the user list with no login and no admin check. All five Claude Code runs named it as their first finding and rated it High. All five Codex runs said do not ship, severity high.

![Actual terminal capture of Claude Code security review (the /security-review skill, run headless with claude -p), Claude Code 2.1.322, model claude-opus-5-5 (Opus 5.5), medium effort, all 5 runs on the planted 5-file shop change (Next.js + Supabase + Stripe, fake key values): the first finding line of runs 1, 2, 3, 4 and 5 each names the unprotected admin route app/api/admin/users/route.ts, so the route was caught 5 of 5 times and every run rated it High. Text rendered from the saved capture files claude-security-review-run1.md to run5.md.](/blog/claude-code-security-review/terminal-claude-code-five-runs-admin-route.webp)

*Actual capture. Claude Code 2.1.322 · runs 1 to 5. Source: claude-security-review-run1.md to claude-security-review-run5.md, line 1 of each (line 7 in run 5) (capture `rulebook-vs-llm-review-v2`).*

![Actual terminal capture of the Codex CLI security review answers, Codex CLI 0.167.1, model gpt-6.1-sol (GPT Sol), reasoning effort medium, read-only sandbox, runs 1 to 5 on the planted 5-file shop change (Next.js + Supabase + Stripe, fake key values): all five runs print VERDICT: DO NOT SHIP and SEVERITY: high, so the verdict and the severity never changed. Text rendered from the saved capture files codex-verbatim-run1.log to codex-verbatim-run5.log, lines 134 to 135.](/blog/claude-code-security-review/terminal-codex-five-runs-do-not-ship.webp)

*Actual capture. Codex CLI 0.167.1 · one-sentence prompt · runs 1 to 5. Source: codex-verbatim-run1.log to codex-verbatim-run5.log, lines 134-135 of each (capture `rulebook-vs-llm-review-v2`).*

That is the strength: a model sees that nothing checks who is asking. A pattern list only matches shapes it knows. Theo says to make the agent review its own pull request before a person sees it:

> If you're putting up PR's that your agent hasn't even reviewed yet, you are wasting the time of your team and it upsets me.

[![Thumbnail of Theo's (t3.gg) YouTube video 'How Claude Fixes It's Own Code'; click to watch the moment at 8:16 where he says: 'If you're putting up PR's that your agent hasn't even reviewed yet, you are wasting the time of your team and it upsets me.'](/blog/claude-code-security-review/thumb-yt-theo.webp)](https://www.youtube.com/watch?v=TFCDWrEBaFM&t=496s)

*Theo (t3.gg), [How Claude Fixes It's Own Code](https://www.youtube.com/watch?v=TFCDWrEBaFM&t=496s), uploaded 2026-11-06, at 8:16.*

### Why does Claude Code security review give a different answer each run?

Run it again and it may say something else. In one run our live key was dismissed as a probable placeholder. In another, the same line said rotate it now.

![Actual terminal capture of Claude Code security review (the /security-review skill, run headless with claude -p), Claude Code 2.1.322, model claude-opus-5-5 (Opus 5.5), medium effort, run 3 of 5 on the planted 5-file shop change (Next.js + Supabase + Stripe, fake key values): the hard-coded live key in lib/stripe.ts line 5 is listed under 'Findings dropped below the confidence threshold of 8' with confidence 7/10 and called a likely false positive, because the review reads the value as a placeholder. Text rendered from the saved capture file claude-security-review-run3.md. Part of the AgenticCLI test of Claude Code, Codex and a fixed-rule scanner on security review.](/blog/claude-code-security-review/terminal-claude-code-run3-live-key-false-positive.webp)

*Actual capture. Claude Code 2.1.322 · run 3 of 5 · result: 1 finding, key dropped · runs 2 to 4 also used claude-sonnet-5-5 inside the skill. Source: claude-security-review-run3.md, lines 18, 28-29 (capture `rulebook-vs-llm-review-v2`).*

![Actual terminal capture of Claude Code security review (the /security-review skill, run headless with claude -p), Claude Code 2.1.322, model claude-opus-5-5 (Opus 5.5), medium effort, run 5 of 5 on the planted 5-file shop change (Next.js + Supabase + Stripe, fake key values): the hard-coded live Stripe key in lib/stripe.ts line 5 is a numbered finding with Severity High and Confidence 9/10, and the recommendation reads 'Revoke and rotate this key in Stripe now', while run 3 called the same line a likely false positive. Text rendered from the saved capture file claude-security-review-run5.md. Part of the AgenticCLI test of Claude Code, Codex and a fixed-rule scanner on security review.](/blog/claude-code-security-review/terminal-claude-code-run5-live-key-rotate-now.webp)

*Actual capture. Claude Code 2.1.322 · run 5 of 5 · result: 3 findings, key reported. Source: claude-security-review-run5.md, lines 18-22, 25 (capture `rulebook-vs-llm-review-v2`).*

A model writes one word at a time and picks with a little chance. Switching the chance off does not fully fix it: your request is batched with other people's, and tiny maths differences can flip a near tie. Thinking Machines Lab sent one prompt a thousand times to one open model with randomness off:

> Surprisingly, we generate 80 unique completions

[![The Thinking Machines Lab post 'Defeating Nondeterminism in LLM Inference' by Horace He and colleagues, 2025-09-10: 'Surprisingly, we generate 80 unique completions, with the most common of these occuring 78 times.'](/blog/claude-code-security-review/shot-web-tml.webp)](https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/)

*Horace He and colleagues, [Defeating Nondeterminism in LLM Inference](https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/), Thinking Machines Lab, 2025-09-10. One prompt, Qwen3-235B, temperature 0.*

On an easy bug only the words move. On a close call the word that moves could be "ship". We did not test a close call.

## What is a rule-based security scanner great at?

A rulebook cannot change its mind, and that is enough to block a push. Each rule is code, so there is no next word to choose. Matt Pocock on tests and linters:

> Deterministic checks, where we have linting and tests and type checking and code quality metrics. All of these things going together, and they all work the same every time.

*Matt Pocock (AI Hero), [Fixing the PR Bottleneck](https://www.youtube.com/watch?v=LlgiOCmFG_w&t=190s), AI Engineer, uploaded 2026-09-25, at 3:10.*

### What did the rulebook do in five runs?

Five runs. Five identical answers. Three blocking findings every time.

We ran ShipGuard 0.11.0 five times, each from a fresh copy of the same tree, telemetry off. Exit code 1 and verdict DO NOT SHIP in all five. The raw text differs by two machine fields, an elapsed time and a run id; strip those and all five outputs hash to the same value, `161ee832c47d`. Same stack as [our Next.js and Supabase scan](/blog/secret-scanner-nextjs-supabase).

![Actual terminal capture of the ShipGuard CLI 0.11.0 rule-based security scan (fixed rules, no model, offline, scan --strict), run 1 of 5 on the planted 5-file shop change (Next.js + Supabase + Stripe, fake key values): CRITICAL, 1 finding: a hardcoded Stripe live secret key (sk_live_ prefix) detected in source at lib/stripe.ts:5, low false-positive chance, evidence masked as sk_live_***. Text rendered from the saved capture file shipguard-full-text-run1.out, lines 3 to 5. Part of the AgenticCLI test of Claude Code, Codex and a fixed-rule scanner.](/blog/claude-code-security-review/terminal-shipguard-live-key-critical.webp)

*Actual capture. ShipGuard CLI 0.11.0 · scan --strict, full project scan · run 1 of 5. Source: shipguard-full-text-run1.out, lines 3-5 (capture `rulebook-vs-llm-review-v2`).*

![Actual terminal capture of the ShipGuard CLI 0.11.0 rule-based security scan (fixed rules, no model, offline, scan --strict), run 1 of 5 on the planted 5-file shop change (Next.js + Supabase + Stripe, fake key values): HIGH, 2 findings: an unprotected API route handler at app/api/admin/users/route.ts and a Math.random() reset token at lib/reset-token.ts:3. Text rendered from the saved capture file shipguard-full-text-run1.out, lines 7 to 10. Part of the AgenticCLI test of Claude Code, Codex and a fixed-rule scanner.](/blog/claude-code-security-review/terminal-shipguard-admin-route-reset-token-high.webp)

*Actual capture. ShipGuard CLI 0.11.0 · scan --strict, full project scan · run 1 of 5. Source: shipguard-full-text-run1.out, lines 7-10 (capture `rulebook-vs-llm-review-v2`).*

![Actual terminal capture of the ShipGuard CLI 0.11.0 rule-based security scan (fixed rules, no model, offline, scan --strict), run 1 of 5 on the planted 5-file shop change (Next.js + Supabase + Stripe, fake key values): Coverage: scanned 14 of 14 files; Total findings: 4 (1 critical, 2 high, 1 low); Blocked: 3 critical/blocking issue(s) must be fixed first; verdict DO NOT SHIP. Text rendered from the saved capture file shipguard-full-text-run1.out, lines 22 to 23, 51 to 52, 55 and 66 to 67. Part of the AgenticCLI test of Claude Code, Codex and a fixed-rule scanner.](/blog/claude-code-security-review/terminal-shipguard-summary-do-not-ship.webp)

*Actual capture. ShipGuard CLI 0.11.0 · scan --strict, full project scan · run 1 of 5. Source: shipguard-full-text-run1.out, lines 22-23, 51-52, 55, 66-67 (capture `rulebook-vs-llm-review-v2`).*

![Actual terminal capture of the version stamp saved with the security review test runs: shipguard 0.11.0, node v24.22.3, git version 2.43.0, time stamp 2026-11-11T23:13:05Z. Text rendered from the saved capture file version-stamp.txt, lines 1 to 4. Part of the AgenticCLI test of Claude Code, Codex and a fixed-rule scanner.](/blog/claude-code-security-review/terminal-version-stamp-shipguard-0-11-0.webp)

*Actual capture. Saved by the capture script before the ShipGuard runs. Source: version-stamp.txt, lines 1-4 (capture `rulebook-vs-llm-review-v2`).*

### What did Claude Code's security review do on the same diff?

Two runs reported three problems. Three runs reported one. "Reported" means a numbered finding in the final report.

Runs 1 and 5 listed the route, the reset token and the live key. Runs 2, 3 and 4 listed the route only. In runs 2 and 3 the key sat in a filtered-out list; in run 4 it was never mentioned. It is the classic leak that [a plain key check](/blog/detect-hardcoded-secrets) exists to catch.

![Actual terminal capture of Claude Code security review (the /security-review skill, run headless with claude -p), Claude Code 2.1.322, model claude-opus-5-5 (Opus 5.5), medium effort, run 1 of 5 on the planted 5-file shop change (Next.js + Supabase + Stripe, fake key values): finding 3 of 3 reads 'Hardcoded live Stripe secret key fallback' in lib/stripe.ts line 5, Severity High, Confidence 8/10, so this run did report the hard-coded live key. Text rendered from the saved capture file claude-security-review-run1.md. Part of the AgenticCLI test of Claude Code, Codex and a fixed-rule scanner on security review.](/blog/claude-code-security-review/terminal-claude-code-run1-live-key-reported.webp)

*Actual capture. Claude Code 2.1.322 · run 1 of 5 · result: 3 findings, key reported. Source: claude-security-review-run1.md, lines 19-23 (capture `rulebook-vs-llm-review-v2`).*

![Actual terminal capture of Claude Code security review (the /security-review skill, run headless with claude -p), Claude Code 2.1.322, model claude-opus-5-5 (Opus 5.5), medium effort, run 4 of 5 on the planted 5-file shop change (Next.js + Supabase + Stripe, fake key values): the report ends after the admin-route recommendation: no second finding, no third finding and no mention of the hard-coded live key. Text rendered from the saved capture file claude-security-review-run4.md. Part of the AgenticCLI test of Claude Code, Codex and a fixed-rule scanner on security review.](/blog/claude-code-security-review/terminal-claude-code-run4-report-ends-no-key.webp)

*Actual capture. Claude Code 2.1.322 · run 4 of 5 · result: 1 finding, key never mentioned · runs 2 to 4 also used claude-sonnet-5-5 inside the skill. Source: claude-security-review-run4.md, lines 12-15 (capture `rulebook-vs-llm-review-v2`).*

### What did Codex do on the same diff?

Same prompt, same verdict, three different lists of problems. Run 1 named the route and the reset token. Run 3 added the live key. Run 5 added the coupon bug too, the only run that did.

![Actual terminal capture of the Codex CLI security review answer, Codex CLI 0.167.1, model gpt-6.1-sol (GPT Sol), reasoning effort medium, read-only sandbox, run 1 of 5 on the planted 5-file shop change (Next.js + Supabase + Stripe, fake key values): VERDICT DO NOT SHIP, SEVERITY high, and a one-sentence REASON that names the unprotected admin endpoint and the Math.random() reset token but not the hard-coded live key. Text rendered from the saved capture file codex-verbatim-run1.log, lines 133 to 136. Part of the AgenticCLI test of Claude Code, Codex and a fixed-rule scanner.](/blog/claude-code-security-review/terminal-codex-run1-route-and-token.webp)

*Actual capture. Codex CLI 0.167.1 · one-sentence prompt · run 1 of 5. Source: codex-verbatim-run1.log, lines 133-136 (capture `rulebook-vs-llm-review-v2`).*

![Actual terminal capture of the Codex CLI security review answer, Codex CLI 0.167.1, model gpt-6.1-sol (GPT Sol), reasoning effort medium, read-only sandbox, run 3 of 5 on the planted 5-file shop change (Next.js + Supabase + Stripe, fake key values): VERDICT DO NOT SHIP, SEVERITY high, and a REASON that names the admin endpoint, insecure randomness for reset tokens and a Stripe live-style secret key fallback. Text rendered from the saved capture file codex-verbatim-run3.log, lines 133 to 136. Part of the AgenticCLI test of Claude Code, Codex and a fixed-rule scanner.](/blog/claude-code-security-review/terminal-codex-run3-live-key-named.webp)

*Actual capture. Codex CLI 0.167.1 · one-sentence prompt · run 3 of 5. Source: codex-verbatim-run3.log, lines 133-136 (capture `rulebook-vs-llm-review-v2`).*

![Actual terminal capture of the Codex CLI security review answer, Codex CLI 0.167.1, model gpt-6.1-sol (GPT Sol), reasoning effort medium, read-only sandbox, run 5 of 5 on the planted 5-file shop change (Next.js + Supabase + Stripe, fake key values): VERDICT DO NOT SHIP, SEVERITY high, and a REASON that names the admin endpoint, insecure reset tokens, unchecked coupon restrictions (the logic bug no rule covers) and a Stripe live-key fallback. Text rendered from the saved capture file codex-verbatim-run5.log, lines 133 to 136. Part of the AgenticCLI test of Claude Code, Codex and a fixed-rule scanner.](/blog/claude-code-security-review/terminal-codex-run5-coupon-bug-named.webp)

*Actual capture. Codex CLI 0.167.1 · one-sentence prompt · run 5 of 5. Source: codex-verbatim-run5.log, lines 133-136 (capture `rulebook-vs-llm-review-v2`).*

### Where does a rulebook stop?

A rulebook only knows the traps someone wrote down. A bad idea sails through, and file scanners can go quiet on what only a reader would see: [four of them returned nothing on a poisoned MCP tool description](/blog/mcp-security-scanner). Dex Horthy, who builds agents for a living, found the same split in production:

> most production agents weren't that agentic at all. They were mostly just software

[![Thumbnail of the YouTube video '12-Factor Agents: Patterns of reliable LLM applications' by Dex Horthy (HumanLayer) at AI Engineer World's Fair; click to watch the moment at 1:53 where he says: 'most production agents weren't that agentic at all. They were mostly just software'](/blog/claude-code-security-review/thumb-yt-dex.webp)](https://www.youtube.com/watch?v=8kMaTybvDUw&t=113s)

*Dex Horthy (HumanLayer), [12-Factor Agents](https://www.youtube.com/watch?v=8kMaTybvDUw&t=113s), AI Engineer World's Fair, uploaded 2025-07-03, at 1:53.*

### Is a CLAUDE.md rule enough to stop Claude Code from pushing secrets?

You wrote never push secrets in CLAUDE.md. Will the agent follow it every single time? Claude Code's [memory documentation](https://code.claude.com/docs/en/memory) says what that file is:

> Claude treats them as context, not enforced configuration. To block an action regardless of what Claude decides, use a PreToolUse hook instead.

Harrison Chase of LangChain, talking about reliable enterprise agents, says the same about prompting:

> if you prompt an agent to do that, great. It might do that like 90% of the time. You don't know what the LLM will do. If you put that in a deterministic kind of like workflow or code, then it will always do that.

[![Thumbnail of the YouTube video '3 ingredients for building reliable enterprise agents' by Harrison Chase (LangChain) at AI Engineer; click to watch the moment at 6:02 where he says: 'if you prompt an agent to do that, great. It might do that like 90% of the time. You don't know what the LLM will do.'](/blog/claude-code-security-review/thumb-yt-chase.webp)](https://www.youtube.com/watch?v=kTnfJszFxCg&t=362s)

*Harrison Chase (LangChain), [3 ingredients for building reliable enterprise agents](https://www.youtube.com/watch?v=kTnfJszFxCg&t=362s), AI Engineer, uploaded 2025-07-23, at 6:02.*

Jason Lemkin learned it the hard way with a Replit agent during a code freeze (not Claude Code, and a scanner would not have stopped it; our [excessive-agency test](/blog/ai-agent-security-risks-excessive-agency) covers that gap):

> There is no way to enforce a code freeze in vibe coding apps like Replit. There just isn’t.

[![Jason Lemkin's post on X, 2025-07-19: 'There is no way to enforce a code freeze in vibe coding apps like Replit. There just isn't.'](/blog/claude-code-security-review/shot-x-jason.webp)](https://x.com/jasonlk/status/1946589071519948952)

*Jason Lemkin, [post on X](https://x.com/jasonlk/status/1946589071519948952), 2025-07-19. The point is only that a written rule is a request.*

## Where do an AI review and a rulebook overlap, and what do both miss?

Each one has a blind spot the other does not.

### Which catches more: Claude Code, Codex or a rule-based scanner?

Four problems planted. No tool caught all four in every run. Counts are runs out of five.

| Planted problem | ShipGuard 0.11.0 | Claude Code · Codex |
|---|---|---|
| Unprotected admin route | 5 of 5 | 5 of 5 · 5 of 5 |
| Weak reset token | 5 of 5 | 2 of 5 · 5 of 5 |
| Hard-coded live key | 5 of 5 | 2 of 5 · 3 of 5 |
| Coupon with no floor (logic bug) | no rule covers it | 0 of 5 · 1 of 5 |

Read it as three zones. Only an AI review can see the coupon bug, where the amount comes off the total with no floor, and it sees it only sometimes. Both can see the route (its service-role client skips [Supabase row-level security](/blog/supabase-rls-ai-built-apps) entirely), the token and the key. And the rulebook adds certainty: the same 3 blocking findings in 5 of 5 runs, including the runs where an AI review stayed quiet.

Asked for every finding, one per line, Codex named the key and the coupon bug in all five runs and still gave a different list each time, with 6, 5, 5, 7 and 5 findings.

![Actual terminal capture of Codex CLI security review with a findings-list prompt, Codex CLI 0.167.1, model gpt-6.1-sol (GPT Sol), reasoning effort medium, read-only sandbox, run 1 of 5 on the planted 5-file shop change (Next.js + Supabase + Stripe, fake key values): item 3 names the hard-coded live-mode secret in lib/stripe.ts rated high, and item 4 names the coupon bug in app/api/checkout/route.ts, a logic problem no rule covers, rated medium. Text rendered from the saved capture file codex-findings-list-run1.log, lines 133 to 134 and 137 to 138.](/blog/claude-code-security-review/terminal-codex-findings-list-key-and-coupon.webp)

*Actual capture. Codex CLI 0.167.1 · findings-list prompt (asks for every finding) · run 1 of 5. Source: codex-findings-list-run1.log, lines 133-134, 137-138 (capture `rulebook-vs-llm-review-v2`).*

### What do builders say about the gap that is left?

Even a green pipeline does not settle it. Matt Pocock:

> Does a green CI mean that the code is ready for merge? No, it does not.

*Matt Pocock (AI Hero), [Fixing the PR Bottleneck](https://www.youtube.com/watch?v=LlgiOCmFG_w&t=293s), AI Engineer, uploaded 2026-09-25, at 4:53.*

Addy Osmani on why rules cannot close that gap:

> you can’t write a test for behavior you haven’t thought to specify.

[![Addy Osmani's blog post 'Comprehension Debt', 2026-03-14: 'you can't write a test for behavior you haven't thought to specify.'](/blog/claude-code-security-review/shot-web-addy-debt.webp)](https://addyosmani.com/blog/comprehension-debt/)

*Addy Osmani, [Comprehension Debt](https://addyosmani.com/blog/comprehension-debt/), 2026-03-14.*

What is left is the part no tool checks: whether the change is the right idea. That is yours.

### What did this test not cover?

Five runs is small. These are counts, not rates.

- A close call. All 10 AI reviews said do not ship, so we did not measure a flip.
- The Sonnet step inside Claude Code's skill in runs 2 to 4. We cannot say whether it caused the spread.
- Codex's one-sentence answers limit what it can name; a findings-list prompt named more.
- The typed `/security-review` returns empty output in headless mode, so we asked Claude Code to run the same skill by prompt.
- Three of the 4 planted problems came from ShipGuard's own rule list, so the test favors it.
- Package names. We did not test whether either review notices a dependency an agent invented; that failure, [slopsquatting](/blog/slopsquatting), has its own four-scanner test.

## What is a setup you can copy today?

Put the rules where your agent cannot skip them. Claude Code runs a hook itself, at a fixed point, every time; its [hooks guide](https://code.claude.com/docs/en/hooks-guide) calls that deterministic control. Our [Claude Code guide](/guides/claude-code) shows ShipGuard running from inside one. Not on Claude Code? A plain git `pre-push` hook does the same job for Codex, or for you. Our [pre-commit comparison](/blog/pre-commit-secret-scanner) shows where each placement misses.

### What is the order we use?

Rules first. Then an AI review. Then you. Matt Pocock describes the same order, and why the first layer is the cheap one:

> Now, automated checks are cheap. That's the cool thing about them is that they don't cost tokens like automated review does, they don't cost human effort, they just cost CPU cycles.

[![Frame from the YouTube video 'Fixing the PR Bottleneck' by Matt Pocock at 4:12, the speaker on stage, taken from the video at the moment he says: 'Now, automated checks are cheap.'](/blog/claude-code-security-review/frame-yt-pocock-4m12.webp)](https://www.youtube.com/watch?v=LlgiOCmFG_w&t=252s)

*Matt Pocock, [Fixing the PR Bottleneck](https://www.youtube.com/watch?v=LlgiOCmFG_w&t=252s), uploaded 2026-09-25, at 4:12.*

When a check fails, the agent reads the finding, fixes it and runs the check again. Boris Cherny, who created Claude Code, says why that loop matters:

> A final tip: probably the most important thing to get great results out of Claude Code -- give Claude a way to verify its work. If Claude has that feedback loop, it will 2-3x the quality of the final result.

[![Boris Cherny's post 13 of a thread on X, 2026-01-02: 'A final tip: probably the most important thing to get great results out of Claude Code -- give Claude a way to verify its work. If Claude has that feedback loop, it will 2-3x the quality of the final result.'](/blog/claude-code-security-review/shot-x-boris13.webp)](https://x.com/bcherny/status/2007179861115511237)

*Boris Cherny, [post 13 of a thread on X](https://x.com/bcherny/status/2007179861115511237), 2026-01-02. The "2-3x" is his estimate, not a benchmark.*

### How do you block git push in Claude Code with a hook?

Boris Cherny points the same way for long-running tasks:

> use an agent Stop hook to do that more deterministically

[![Boris Cherny's post 12 of a thread on X, 2026-01-02: 'For very long-running tasks, I will either (a) prompt Claude to verify its work with a background agent when it's done, (b) use an agent Stop hook to do that more deterministically, or (c) use the ralph-wiggum plugin.'](/blog/claude-code-security-review/shot-x-boris12.webp)](https://x.com/bcherny/status/2007179858435281082)

*Boris Cherny, [post 12 of the same thread](https://x.com/bcherny/status/2007179858435281082), 2026-01-02.*

Two small pieces stand between your agent and a bad push: a script that wakes only on `git push`, runs a checker and exits 2 on a finding, and a `PreToolUse` entry in `.claude/settings.json` (matcher `Bash`) that points at it. This is the part that does the blocking, from `.claude/hooks/pre-push-check.sh`:

```diff
+++ .claude/hooks/pre-push-check.sh
+cmd=$(jq -r '.tool_input.command // empty')
+case "$cmd" in *"git push"*) ;; *) exit 0 ;; esac
+
+if ! out=$(eval "$CHECK" 2>&1); then
+  echo "Push blocked: the pre-push check found something." >&2
+  exit 2
+fi
```

`CHECK` is any command that exits non-zero on a finding. The [hooks guide](https://code.claude.com/docs/en/hooks-guide) shows the settings entry in full.

We ran it on Claude Code 2.1.322, headless, asking it to run `git push origin HEAD`. With a checker set to fail, the push was stopped and nothing reached the remote. With a checker set to pass, the push went through. Our checker was a stand-in, not a real secret scanner ([gitleaks, TruffleHog and ShipGuard on one repo](/blog/github-secret-scanner-vs-cli-tools) shows how real ones differ), so treat the checker you choose as a starting point.

![Actual terminal capture of Claude Code 2.1.322 reporting that its PreToolUse hook blocked git push origin HEAD on the feature/admin-export branch: the pre-push hook stopped it with 'Push blocked: the pre-push check found something', so nothing went to origin. Text rendered from the saved capture file claude-answer-blocked.txt of the hook test. Part of the AgenticCLI test of CLAUDE.md rules versus hooks.](/blog/claude-code-security-review/terminal-claude-code-hook-blocked-git-push.webp)

*Actual capture. Claude Code 2.1.322, run headless · a PreToolUse hook on Bash runs a stand-in check; the check was set to fail · only Claude's final answer text was saved, not the hook's own message. Source: claude-answer-blocked.txt, line 1 (capture `rulebook-vs-llm-review`).*

## What should a rulebook gate do?

If it can change its mind, it is not a gate. Five points:

1. **Same answer every run.** The whole reason to have one.
2. **Runs where your agent works.** A hook for Claude Code, a git `pre-push` hook for Codex, and CI as the door nobody can skip.
3. **Knows the common traps.** A key in a file git will commit, a route with no login, a migration that drops a table. Our [twelve-check guide](/blog/vibe-coding-security-12-checks) lists more.
4. **Cheap enough that nobody skips it.** A slow gate gets bypassed, and an agent bypasses it first.
5. **Says what to fix.** Our stand-in printed no reason, and Claude Code answered: "The hook didn't say what it found, so the next step is to open that script and see what it checks."

## Where does ShipGuard fit?

We're AgenticCLI. We build tools for agentic coding, including ShipGuard: a security scanner with a rulebook and a hard gate. It runs on your machine. Once you set it up as a hook or a CI step, your agent runs it before every commit.

Try it now *for free*:

```bash
npx @agenticcli/shipguard scan
```

Add `--strict` and it exits 1 on a blocking finding, so a hook or CI step can stop on it. It reports what its rules cover and nothing else. It would not have caught the coupon bug, and it does not judge whether a change is the right idea. Pair it with a review that can read, and with your own eyes. See [what each free check covers](/shipguard), and the [docs](/shipguard/docs).

## Sources

Every number above comes from our saved run files: raw logs for all 15 runs (five per tool), stamped with the tool versions, in capture `rulebook-vs-llm-review-v2`, plus the hook test in capture `rulebook-vs-llm-review`. Run logs are kept on file, not published.

- Matt Pocock (AI Hero), [Fixing the PR Bottleneck](https://www.youtube.com/watch?v=LlgiOCmFG_w&t=140s), AI Engineer, 2026-09-25.
- Theo (t3.gg), [How Claude Fixes It's Own Code](https://www.youtube.com/watch?v=TFCDWrEBaFM&t=496s), 2026-11-06.
- Dex Horthy (HumanLayer), [12-Factor Agents](https://www.youtube.com/watch?v=8kMaTybvDUw&t=113s), 2025-07-03.
- Harrison Chase (LangChain), [3 ingredients for building reliable enterprise agents](https://www.youtube.com/watch?v=kTnfJszFxCg&t=362s), 2025-07-23.
- Addy Osmani, [Agentic Code Review](https://addyosmani.com/blog/agentic-code-review/) and [Comprehension Debt](https://addyosmani.com/blog/comprehension-debt/).
- Thinking Machines Lab, [Defeating Nondeterminism in LLM Inference](https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/), 2025-09-10.
- X posts: Boris Cherny, [post 12](https://x.com/bcherny/status/2007179858435281082) and [post 13](https://x.com/bcherny/status/2007179861115511237); Jason Lemkin, [post](https://x.com/jasonlk/status/1946589071519948952).
- Anthropic, Claude Code [commands](https://code.claude.com/docs/en/commands), [memory](https://code.claude.com/docs/en/memory) and [hooks guide](https://code.claude.com/docs/en/hooks-guide).
