Claude Code or Codex for Security Review? We Tested It
On this page0 / 8
- Why does everyone want an AI to do the reading?
- What are Claude Code and Codex security reviews great at?
- What is a rule-based security scanner great at?
- Where do an AI review and a rulebook overlap, and what do both miss?
- What is a setup you can copy today?
- What should a rulebook gate do?
- Where does ShipGuard fit?
- Sources
Short answer: Claude Code and Codex read your change like a colleague, and both named the open admin route every run. But Claude Code gave 2 different answers in five runs, and 3 runs left a hard-coded live key out. A fixed rulebook reported the same 3 problems 5 of 5 times. Use both: rules first, an AI review second, you last.
A live key sat in the code. We asked Claude Code's security review to look, five times, on the same change. Three times it left the key out of its findings.
The key is a fake value in the real live-key format, planted on purpose in a small Next.js and Supabase shop. Codex, run five times on the same change, left it out twice. A fixed rulebook reported it in all 5 runs.
Nobody was wrong about the verdict: all 10 AI reviews said do not ship. What moved was which problems got named. Ship on the quiet run and you ship the key.
These are our own runs, in our own terminal. The terminal windows below are real output, not mock-ups. Circled numbers mark the lines in question. Spoken quotes come from YouTube's auto-generated captions.
How we tested. Same machine, five runs each, one 5-file change with four planted problems: a hard-coded live key, an admin route with no login, a weak password-reset token and a coupon that can push a total below zero.
| Tool | Version | Model and setting |
|---|---|---|
| ShipGuard CLI | 0.11.0 | fixed rules, offline, no model |
Claude Code /security-review |
2.1.322 | claude-opus-5-5, medium effort |
| Codex CLI | 0.167.1 | gpt-6.1-sol, medium effort, read-only |
In runs 2, 3 and 4 the /security-review skill also ran a claude-sonnet-5-5 step inside it; those are the three runs that left the key out. Runs 1 and 5 used Opus only. We cannot say whether that step is the cause. And Codex answered a one-sentence prompt, which limits what it can name. With a findings-list prompt it named the live key and the coupon bug in 5 of 5 runs.
Why does everyone want an AI to do the reading?
Your agent opens pull requests faster than you can read them. So you hand the reading to an AI. What did you just stop checking?
Matt Pocock named the pressure in his AI Engineer talk on the pull request bottleneck:
If you just have permanent acceleration pushing stuff through your factory, you're going to end up with a slop cannon, right?
Matt Pocock (AI Hero), Fixing the PR Bottleneck, AI Engineer, uploaded 2026-09-25, at 2:21.
Addy Osmani saw what teams did next:
Reviewers simply could not keep pace with the volume, so code began merging unread, and that became normal.
His one figure comes from Faros AI, a vendor selling into this market, so read it with that in mind:
PRs merged with zero review up 31.3%
Addy Osmani, Agentic Code Review, 2026-06-15.
Handing the reading to a model is reasonable. Our agent security guide covers how much an agent may do alone. Here: what does the model read well?
What are Claude Code and Codex security reviews great at?
It reads your change the way a colleague would. Anthropic's command reference describes /security-review like this:
Analyze the changes on your current branch for security vulnerabilities. Reviews the diff between your branch and origin’s default branch, identifying risks like injection, auth issues, and data exposure.
Codex does the same job when you ask in plain words. Both read the diff and ask what the change is for. A rulebook does neither. For what Claude Code writes in the first place, see our audit of its output.
Does it understand what a change is for?
One line opened an admin route that exports every customer. Ten of ten AI reviews caught it.
The route, app/api/admin/users/route.ts, returns the user list with no login and no admin check. All five Claude Code runs named it as their first finding and rated it High. All five Codex runs said do not ship, severity high.

Actual capture. Claude Code 2.1.322 · runs 1 to 5. Source: claude-security-review-run1.md to claude-security-review-run5.md, line 1 of each (line 7 in run 5) (capture rulebook-vs-llm-review-v2).

Actual capture. Codex CLI 0.167.1 · one-sentence prompt · runs 1 to 5. Source: codex-verbatim-run1.log to codex-verbatim-run5.log, lines 134-135 of each (capture rulebook-vs-llm-review-v2).
That is the strength: a model sees that nothing checks who is asking. A pattern list only matches shapes it knows. Theo says to make the agent review its own pull request before a person sees it:
If you're putting up PR's that your agent hasn't even reviewed yet, you are wasting the time of your team and it upsets me.
Theo (t3.gg), How Claude Fixes It's Own Code, uploaded 2026-11-06, at 8:16.
Why does Claude Code security review give a different answer each run?
Run it again and it may say something else. In one run our live key was dismissed as a probable placeholder. In another, the same line said rotate it now.

Actual capture. Claude Code 2.1.322 · run 3 of 5 · result: 1 finding, key dropped · runs 2 to 4 also used claude-sonnet-5-5 inside the skill. Source: claude-security-review-run3.md, lines 18, 28-29 (capture rulebook-vs-llm-review-v2).

Actual capture. Claude Code 2.1.322 · run 5 of 5 · result: 3 findings, key reported. Source: claude-security-review-run5.md, lines 18-22, 25 (capture rulebook-vs-llm-review-v2).
A model writes one word at a time and picks with a little chance. Switching the chance off does not fully fix it: your request is batched with other people's, and tiny maths differences can flip a near tie. Thinking Machines Lab sent one prompt a thousand times to one open model with randomness off:
Surprisingly, we generate 80 unique completions
Horace He and colleagues, Defeating Nondeterminism in LLM Inference, Thinking Machines Lab, 2025-09-10. One prompt, Qwen3-235B, temperature 0.
On an easy bug only the words move. On a close call the word that moves could be "ship". We did not test a close call.
What is a rule-based security scanner great at?
A rulebook cannot change its mind, and that is enough to block a push. Each rule is code, so there is no next word to choose. Matt Pocock on tests and linters:
Deterministic checks, where we have linting and tests and type checking and code quality metrics. All of these things going together, and they all work the same every time.
Matt Pocock (AI Hero), Fixing the PR Bottleneck, AI Engineer, uploaded 2026-09-25, at 3:10.
What did the rulebook do in five runs?
Five runs. Five identical answers. Three blocking findings every time.
We ran ShipGuard 0.11.0 five times, each from a fresh copy of the same tree, telemetry off. Exit code 1 and verdict DO NOT SHIP in all five. The raw text differs by two machine fields, an elapsed time and a run id; strip those and all five outputs hash to the same value, 161ee832c47d. Same stack as our Next.js and Supabase scan.

Actual capture. ShipGuard CLI 0.11.0 · scan --strict, full project scan · run 1 of 5. Source: shipguard-full-text-run1.out, lines 3-5 (capture rulebook-vs-llm-review-v2).

Actual capture. ShipGuard CLI 0.11.0 · scan --strict, full project scan · run 1 of 5. Source: shipguard-full-text-run1.out, lines 7-10 (capture rulebook-vs-llm-review-v2).

Actual capture. ShipGuard CLI 0.11.0 · scan --strict, full project scan · run 1 of 5. Source: shipguard-full-text-run1.out, lines 22-23, 51-52, 55, 66-67 (capture rulebook-vs-llm-review-v2).

Actual capture. Saved by the capture script before the ShipGuard runs. Source: version-stamp.txt, lines 1-4 (capture rulebook-vs-llm-review-v2).
What did Claude Code's security review do on the same diff?
Two runs reported three problems. Three runs reported one. "Reported" means a numbered finding in the final report.
Runs 1 and 5 listed the route, the reset token and the live key. Runs 2, 3 and 4 listed the route only. In runs 2 and 3 the key sat in a filtered-out list; in run 4 it was never mentioned. It is the classic leak that a plain key check exists to catch.

Actual capture. Claude Code 2.1.322 · run 1 of 5 · result: 3 findings, key reported. Source: claude-security-review-run1.md, lines 19-23 (capture rulebook-vs-llm-review-v2).

Actual capture. Claude Code 2.1.322 · run 4 of 5 · result: 1 finding, key never mentioned · runs 2 to 4 also used claude-sonnet-5-5 inside the skill. Source: claude-security-review-run4.md, lines 12-15 (capture rulebook-vs-llm-review-v2).
What did Codex do on the same diff?
Same prompt, same verdict, three different lists of problems. Run 1 named the route and the reset token. Run 3 added the live key. Run 5 added the coupon bug too, the only run that did.

Actual capture. Codex CLI 0.167.1 · one-sentence prompt · run 1 of 5. Source: codex-verbatim-run1.log, lines 133-136 (capture rulebook-vs-llm-review-v2).

Actual capture. Codex CLI 0.167.1 · one-sentence prompt · run 3 of 5. Source: codex-verbatim-run3.log, lines 133-136 (capture rulebook-vs-llm-review-v2).

Actual capture. Codex CLI 0.167.1 · one-sentence prompt · run 5 of 5. Source: codex-verbatim-run5.log, lines 133-136 (capture rulebook-vs-llm-review-v2).
Where does a rulebook stop?
A rulebook only knows the traps someone wrote down. A bad idea sails through, and file scanners can go quiet on what only a reader would see: four of them returned nothing on a poisoned MCP tool description. Dex Horthy, who builds agents for a living, found the same split in production:
most production agents weren't that agentic at all. They were mostly just software
Dex Horthy (HumanLayer), 12-Factor Agents, AI Engineer World's Fair, uploaded 2025-07-03, at 1:53.
Is a CLAUDE.md rule enough to stop Claude Code from pushing secrets?
You wrote never push secrets in CLAUDE.md. Will the agent follow it every single time? Claude Code's memory documentation says what that file is:
Claude treats them as context, not enforced configuration. To block an action regardless of what Claude decides, use a PreToolUse hook instead.
Harrison Chase of LangChain, talking about reliable enterprise agents, says the same about prompting:
if you prompt an agent to do that, great. It might do that like 90% of the time. You don't know what the LLM will do. If you put that in a deterministic kind of like workflow or code, then it will always do that.
Harrison Chase (LangChain), 3 ingredients for building reliable enterprise agents, AI Engineer, uploaded 2025-07-23, at 6:02.
Jason Lemkin learned it the hard way with a Replit agent during a code freeze (not Claude Code, and a scanner would not have stopped it; our excessive-agency test covers that gap):
There is no way to enforce a code freeze in vibe coding apps like Replit. There just isn’t.
Jason Lemkin, post on X, 2025-07-19. The point is only that a written rule is a request.
Where do an AI review and a rulebook overlap, and what do both miss?
Each one has a blind spot the other does not.
Which catches more: Claude Code, Codex or a rule-based scanner?
Four problems planted. No tool caught all four in every run. Counts are runs out of five.
| Planted problem | ShipGuard 0.11.0 | Claude Code · Codex |
|---|---|---|
| Unprotected admin route | 5 of 5 | 5 of 5 · 5 of 5 |
| Weak reset token | 5 of 5 | 2 of 5 · 5 of 5 |
| Hard-coded live key | 5 of 5 | 2 of 5 · 3 of 5 |
| Coupon with no floor (logic bug) | no rule covers it | 0 of 5 · 1 of 5 |
Read it as three zones. Only an AI review can see the coupon bug, where the amount comes off the total with no floor, and it sees it only sometimes. Both can see the route (its service-role client skips Supabase row-level security entirely), the token and the key. And the rulebook adds certainty: the same 3 blocking findings in 5 of 5 runs, including the runs where an AI review stayed quiet.
Asked for every finding, one per line, Codex named the key and the coupon bug in all five runs and still gave a different list each time, with 6, 5, 5, 7 and 5 findings.

Actual capture. Codex CLI 0.167.1 · findings-list prompt (asks for every finding) · run 1 of 5. Source: codex-findings-list-run1.log, lines 133-134, 137-138 (capture rulebook-vs-llm-review-v2).
What do builders say about the gap that is left?
Even a green pipeline does not settle it. Matt Pocock:
Does a green CI mean that the code is ready for merge? No, it does not.
Matt Pocock (AI Hero), Fixing the PR Bottleneck, AI Engineer, uploaded 2026-09-25, at 4:53.
Addy Osmani on why rules cannot close that gap:
you can’t write a test for behavior you haven’t thought to specify.
Addy Osmani, Comprehension Debt, 2026-03-14.
What is left is the part no tool checks: whether the change is the right idea. That is yours.
What did this test not cover?
Five runs is small. These are counts, not rates.
- A close call. All 10 AI reviews said do not ship, so we did not measure a flip.
- The Sonnet step inside Claude Code's skill in runs 2 to 4. We cannot say whether it caused the spread.
- Codex's one-sentence answers limit what it can name; a findings-list prompt named more.
- The typed
/security-reviewreturns empty output in headless mode, so we asked Claude Code to run the same skill by prompt. - Three of the 4 planted problems came from ShipGuard's own rule list, so the test favors it.
- Package names. We did not test whether either review notices a dependency an agent invented; that failure, slopsquatting, has its own four-scanner test.
What is a setup you can copy today?
Put the rules where your agent cannot skip them. Claude Code runs a hook itself, at a fixed point, every time; its hooks guide calls that deterministic control. Our Claude Code guide shows ShipGuard running from inside one. Not on Claude Code? A plain git pre-push hook does the same job for Codex, or for you. Our pre-commit comparison shows where each placement misses.
What is the order we use?
Rules first. Then an AI review. Then you. Matt Pocock describes the same order, and why the first layer is the cheap one:
Now, automated checks are cheap. That's the cool thing about them is that they don't cost tokens like automated review does, they don't cost human effort, they just cost CPU cycles.
Matt Pocock, Fixing the PR Bottleneck, uploaded 2026-09-25, at 4:12.
When a check fails, the agent reads the finding, fixes it and runs the check again. Boris Cherny, who created Claude Code, says why that loop matters:
A final tip: probably the most important thing to get great results out of Claude Code -- give Claude a way to verify its work. If Claude has that feedback loop, it will 2-3x the quality of the final result.
Boris Cherny, post 13 of a thread on X, 2026-01-02. The "2-3x" is his estimate, not a benchmark.
How do you block git push in Claude Code with a hook?
Boris Cherny points the same way for long-running tasks:
use an agent Stop hook to do that more deterministically
Boris Cherny, post 12 of the same thread, 2026-01-02.
Two small pieces stand between your agent and a bad push: a script that wakes only on git push, runs a checker and exits 2 on a finding, and a PreToolUse entry in .claude/settings.json (matcher Bash) that points at it. This is the part that does the blocking, from .claude/hooks/pre-push-check.sh:
CHECK is any command that exits non-zero on a finding. The hooks guide shows the settings entry in full.
We ran it on Claude Code 2.1.322, headless, asking it to run git push origin HEAD. With a checker set to fail, the push was stopped and nothing reached the remote. With a checker set to pass, the push went through. Our checker was a stand-in, not a real secret scanner (gitleaks, TruffleHog and ShipGuard on one repo shows how real ones differ), so treat the checker you choose as a starting point.

Actual capture. Claude Code 2.1.322, run headless · a PreToolUse hook on Bash runs a stand-in check; the check was set to fail · only Claude's final answer text was saved, not the hook's own message. Source: claude-answer-blocked.txt, line 1 (capture rulebook-vs-llm-review).
What should a rulebook gate do?
If it can change its mind, it is not a gate. Five points:
- Same answer every run. The whole reason to have one.
- Runs where your agent works. A hook for Claude Code, a git
pre-pushhook for Codex, and CI as the door nobody can skip. - Knows the common traps. A key in a file git will commit, a route with no login, a migration that drops a table. Our twelve-check guide lists more.
- Cheap enough that nobody skips it. A slow gate gets bypassed, and an agent bypasses it first.
- Says what to fix. Our stand-in printed no reason, and Claude Code answered: "The hook didn't say what it found, so the next step is to open that script and see what it checks."
Where does ShipGuard fit?
We're AgenticCLI. We build tools for agentic coding, including ShipGuard: a security scanner with a rulebook and a hard gate. It runs on your machine. Once you set it up as a hook or a CI step, your agent runs it before every commit.
Try it now for free:
Add --strict and it exits 1 on a blocking finding, so a hook or CI step can stop on it. It reports what its rules cover and nothing else. It would not have caught the coupon bug, and it does not judge whether a change is the right idea. Pair it with a review that can read, and with your own eyes. See what each free check covers, and the docs.
Sources
Every number above comes from our saved run files: raw logs for all 15 runs (five per tool), stamped with the tool versions, in capture rulebook-vs-llm-review-v2, plus the hook test in capture rulebook-vs-llm-review. Run logs are kept on file, not published.
- Matt Pocock (AI Hero), Fixing the PR Bottleneck, AI Engineer, 2026-09-25.
- Theo (t3.gg), How Claude Fixes It's Own Code, 2026-11-06.
- Dex Horthy (HumanLayer), 12-Factor Agents, 2025-07-03.
- Harrison Chase (LangChain), 3 ingredients for building reliable enterprise agents, 2025-07-23.
- Addy Osmani, Agentic Code Review and Comprehension Debt.
- Thinking Machines Lab, Defeating Nondeterminism in LLM Inference, 2025-09-10.
- X posts: Boris Cherny, post 12 and post 13; Jason Lemkin, post.
- Anthropic, Claude Code commands, memory and hooks guide.
FAQ
Can you rely on Claude Code or Codex for security review? We tested it.
Not to give the same answer every time. We ran Claude Code security review and Codex security review 5 times each on the same 5-file change. Claude Code 2.1.322 gave 2 different answers: 3 runs named only the admin route, 2 runs named 3 problems. Codex 0.167.1 gave 3 different answers. All 10 runs said do not ship, so the verdict held. The list of problems moved, so a clean-looking review is not proof of a clean change. A rulebook gave the same answer 5 times out of 5.
Does Claude Code security review stop a PR with a hard-coded API key? We tested it.
Not reliably. In 5 runs, Claude Code security review (/security-review, Claude Code 2.1.322, Opus 5.5, medium effort) listed our fake live Stripe key (lib/stripe.ts, line 5) as a finding in 2 runs. In runs 2 and 3 it put the key under filtered out, as a possible placeholder. In run 4 it never mentioned it. Runs 2 to 4 also ran a Sonnet 5.5 step inside the skill. The review only writes a report. A rulebook reported the key 5 times out of 5.
Claude Code vs Codex vs a rule-based scanner: which wins at security review consistently and reliably? We tested it.
For AI code review security checks, the rule-based scanner was the most steady: the same 3 problems in 5 of 5 runs. Codex named the live key in 3 of 5 runs and the weak reset token in 5 of 5. Claude Code named each in 2 of 5. Only Codex named the coupon bug no rule covers, in 1 of 5 runs. Asked for a list, Codex named the key and the coupon bug in 5 of 5. Three of the 4 problems came from the scanner's own rule list, so the test favors it.
Is a CLAUDE.md rule enough to stop Claude Code from pushing secrets?
No. Claude Code's docs say CLAUDE.md is context, not enforced configuration, and point to hooks for blocking an action. A hook can block git push in Claude Code. We tested a hook on Claude Code 2.1.322. When the check failed, git push never reached the remote and Claude reported the block. When the check passed, the push went through. A hook is a script Claude Code runs at a fixed point, so it does not depend on the model remembering a line. Our checker was a stand-in, not a real secret scanner.
Don't ship the next one.
Free, local, no account. Catches this exact bug class before deploy.
Quick start: run ShipGuard locally →Compare ShipGuard with Snyk and Semgrep →






