Vetting an AI Agent Framework: We Probed 4 SDKs

19 min · jul 2026

Four agent frameworks went into four empty directories on 28 July 2026, each with its own npm install, and each was handed the same tool: a function called execute_sql that takes one string. Then the same two strings, in the same order. SELECT id FROM tasks LIMIT 10, and DROP TABLE users.

@langchain/core 1.2.3 ran both. Its DynamicStructuredTool carries no member with a name anywhere near approve, authorize, confirm or guard, so there was no step to reach and nothing to intercept. Two SQL strings offered, two executed.

@openai/agents 0.14.0 held the second one, once we gave it a policy of eight words. So did the Vercel AI SDK. That difference is a framework-selection decision, it is checkable before you write a line of application code, and almost nothing written about agent framework security tells you how to check it.

Real capture. @openai/[email protected], [email protected], @anthropic-ai/[email protected], @langchain/[email protected], 28 July 2026. Fixture and manifest: content-pipeline/captures/agent-framework-approval-probe/.

The same four rows, transcribed off that capture:

Can a human stop a tool call after the model picks the arguments?

That is the question worth leading a vetting checklist with, because it is the one whose answer differs between packages and the one a comparison table on a vendor site will never contain. It is also narrower than it sounds. Nobody is asking whether the framework is secure. The question is whether a specific enforcement point exists, and whether it can see what it needs to see.

Three of the four expose one. The OpenAI Agents SDK documents it in the file every install carries:

"Whether the tool needs human approval before it can be called. If this is true, the run will result in an interruption that the program has to resolve by approving or rejecting the tool call."

— OpenAI, @openai/agents-core 0.14.0, dist/tool.d.ts

The Anthropic Claude Agent SDK puts the same idea at the session level rather than on the tool, where one handler sees every call the agent makes.

"Custom permission handler for controlling tool usage. Called before each tool execution to determine if it should be allowed, denied, or prompt the user."

— Anthropic, @anthropic-ai/claude-agent-sdk 0.3.220, sdk.d.ts

What does the hook actually receive?

This is the part that decides whether the hook is worth anything, and it is one line of a type declaration away. ToolApprovalFunction in the OpenAI SDK takes the run context and the parsed input. CanUseTool in the Claude Agent SDK takes (toolName: string, input: Record<string, unknown>, options). Both see the values.

A hook handed nothing but a tool name would be a second capability gate wearing a costume. You already decided which tools exist when you wrote the list, and deciding again with no new information adds a prompt and no judgement.

An academic audit published in June 2026 as arXiv:2606.28679 put that distinction at the centre of its title, and asked of LangChain, LlamaIndex and the Stripe Agent Toolkit whether each one re-authorizes a call with its concrete argument values before execution.

Our probe asks a narrower question of four different packages, and the answers are not the same shape. The pre-ship gate an autonomous agent needs can therefore be written in the framework rather than bolted beside it, which was not true a year ago.

What happens when nobody sets one?

The hook is there and it is off. Calling tool() in the OpenAI SDK without supplying a policy, then asking it about DROP TABLE users, returns false. The same construction in the Vercel AI SDK leaves the property undefined. Neither is a defect; both are the sane default for a library that cannot know which of your calls are dangerous.

It does mean the count that matters for a vetting exercise is not three. It is zero, in the sense that no framework we installed will stop anything until somebody writes a policy, and writing it is not on any getting-started page we found. A permissive default that nobody revisits is the same trap Firebase test mode sets in a different corner of the same problem.

Where is the Vercel SDK moving its enforcement point?

The Vercel SDK adds a wrinkle worth knowing before you build on it. Its tool-level option works today and its own type declaration says where it is going:

"@deprecated Tool approval is handled on a generateText / streamText level now."

— Vercel, @ai-sdk/provider-utils 5.0.14, dist/index.d.ts

That single tag is more useful than a changelog entry, because it tells you the enforcement point is moving up a level and your policy will need to move with it. It also happens to be the kind of fact that only exists in the package. No comparison article contains it.

We did not take our own word for the tag either. The probe greps that shipped declaration for that exact sentence on every run and prints the boolean it gets back, so the day Vercel drops the tag our capture stops saying true without anybody remembering to check.

Which of the four grants does a framework actually check?

Registering a tool is usually described as one action. It is four, and separating them makes the rest of this straightforward, because each one has a different owner.

Two of the four are settled the moment you write the code, and they are the two every framework handles well. The third is the one the approval hook exists for. The fourth belongs to your database and your cloud account, and no package will ever reach it.

The fourth row is the one no package can help with, and it is where the excessive-agency checklist earns its keep. If the connection string in your agent's config is a database owner, an approved query and an unapproved one both arrive at a session that can drop anything. Narrowing that grant is a database-side decision, the same discipline row-level security asks for, and no amount of framework configuration substitutes for it.

A paper submitted in March 2026 puts the missing layer more memorably than we can:

"AI agents today have passwords but no permission slips. They execute tool calls (fund transfers, database queries, shell commands, sub-agent delegation) with no standard mechanism to enforce authorization before the action executes."

— Uchi Uchibeke, arXiv:2603.20953, 21 March 2026

The paper's own testbed is the part worth borrowing for an argument with a colleague. Across 4,437 authorization decisions in 1,151 sessions, social engineering succeeded against the model 74.6% of the time under a permissive policy, and a comparable population of attackers reached a 0% success rate across 879 attempts under a restrictive one. The model did not get better. The policy layer did the work.

What did the install put on the machine?

The second vetting question has nothing to do with agents. Whatever ends up in node_modules runs on the developer machine that also holds the deploy keys, so a framework's dependency tree is part of what you are choosing. Not one of the twenty-one page-one results for this topic installs anything, so not one of them is in a position to ask.

The measurement is cheap. Install each framework alone into an empty directory, walk what arrives, and read every package's own manifest for the three script names npm executes at install time.

Real capture. @openai/[email protected], [email protected], @anthropic-ai/[email protected], @langchain/[email protected], 28 July 2026. Fixture and manifest: content-pipeline/captures/agent-framework-install-surface/.

What surprised us in that table?

Two things in that table were not what we expected. The trees differ by an order of magnitude, from ten packages for the Vercel SDK to a hundred and two for both the OpenAI and Anthropic SDKs, with the Anthropic tree occupying 288 MB. And not one package in any of them declares a preinstall, install or postinstall script.

Dependency weight is not a security metric on its own, and a fat tree from a maintained project beats a slim one from an abandoned account. It does change what you are agreeing to review, which is the same argument that applies to what an AI coding tool leaves in your repository.

What does an empty install-script column buy you?

Less than it sounds, and more than nothing. It is a genuine, checkable null result: the four trees we measured hand npm no opportunity to run arbitrary code on your machine during installation, which is the mechanism behind a good share of the npm compromises of the last two years.

What it does not buy is safety at import time or at run time, where a compromised release does its work anyway. The postmark-mcp backdoor arrived in a routine version bump with no install script involved. Treat this column as one closed door in a building with several.

Worth stating precisely, because we nearly got it wrong ourselves: this is a count of declarations, cross-checked with a plain grep over every package.json in all four trees. Our npm has ignore-scripts set, so it would not have executed one regardless. The durable fact is that there was nothing to execute. Nobody had fitted that door in the first place.

What has the framework already had to disclose?

Checking a project's security history is standard advice. The twenty-one page-one results for this topic carry no version number and no package name between them, so there is nothing on that page for the advice to be about. It is one query per package against the GitHub Advisory Database.

Real capture. GitHub Advisory Database via the REST API, 28 July 2026. Fixture and manifest: content-pipeline/captures/agent-framework-advisory-history/.

The @modelcontextprotocol/sdk row is on that list deliberately, because most agent projects acquire it without choosing it, and the tool servers it connects are a second vetting surface with its own rules.

How should the count be read?

Read that column as a ranking and you will pick the newest package on the list, which is not a security strategy. The two empty rows belong to SDKs that are months old. The twenty-four belongs to a project that has been in production since 2022, and every one of those records is a defect somebody found, reported and fixed, which is what a healthy disclosure process looks like from the outside.

Two other rows repay a second look. @langchain/core carries one of its own, rated high, separate from the twenty-four against langchain — so the package you actually import and the package the number is usually quoted about are different rows. llama-index sits at nine. What that column measures is attention, and attention is a thing you want a dependency to have had.

Why is a path-prefix check the interesting one?

Because it is the kind of bug that lives in a framework rather than in your code, and reading the advisory teaches you what to grep for in the next framework. GHSA-gr75-jv2w-4656, CVE-2026-55443, published on 16 June 2026:

"a file-search agent middleware that validates a starting directory but not the search pattern or the resolved target of matched files, so glob patterns and symlinks can reach files outside the configured root"

— GitHub Advisory Database, GHSA-gr75-jv2w-4656, 16 June 2026

The advisory names the untrusted input source as including an LLM acting on untrusted input, states that there is no evidence of the behaviour being triggered in the wild, and describes a second defect in the same family: an authorization check comparing paths by string prefix with no path-segment boundary, so a sibling directory sharing the prefix is accepted. Validate before resolution and you have validated a different string from the one that gets opened.

What does "treat prompts as executable configuration" mean here?

It means a convenience feature can be a trust boundary, and the framework is where that decision was already made for you. CVE-2026-45134 covers a LangSmith SDK method that pulled a public prompt by owner/name and deserialized the manifest it got back. A manifest can set a model client's base URL.

"Organizations should treat prompts as executable configuration and apply the same review and audit practices they would apply to application code."

— GitHub Advisory Database, GHSA-3644-q5cj-c5c7, 13 May 2026

The remediation is a flag called dangerously_pull_public_prompt, which is the correct name and a good habit to look for. When a framework's fix is to make you type the word dangerous, it has told you where its trust boundary sits.

What does that search turn up in a real install?

Real capture. @openai/[email protected], [email protected], @anthropic-ai/[email protected], @langchain/[email protected], 28 July 2026. Fixture and manifest: content-pipeline/captures/agent-framework-dangerous-options/.

The JavaScript spelling of that same flag, dangerouslyPullPublicPrompt, is already sitting in the @langchain/core tree. It is declared by langsmith, at dist/client.d.ts:1457. Nobody picks langsmith; it arrives underneath. The Claude Agent SDK marks a boundary of its own the same way, at sdk-tools.d.ts:552, where the option decides whether a command may leave the sandbox at all.

The empty row deserves care. This check finds a trust boundary only where a maintainer chose to mark it with this naming convention, so an empty result is a fact about the convention rather than about the package. A null result from a naming search and a null result from a run are not the same kind of quiet.

Will a scanner tell you if you skipped the hook?

Vetting a framework is easier if you know in advance which parts of the answer your existing tooling will hand you. For most teams that tooling is a secret scanner running in CI, and the honest question is whether it has anything at all to say about a tool grant.

We wanted our own answer to this rather than an assurance, so we built the smallest experiment that could produce one. An ops assistant on @openai/agents, copied twice, with src/tools.ts in the second copy swapped for a version where every tool carries a needsApproval policy.

Eleven lines differ between the two files. Two of them, from execute_sql at line 15 and delete_file at line 23 of fixture/src/tools.gated.ts:

needsApproval: async (_ctx, input) => destructive.test(input.query),
needsApproval: async (_ctx, input) => !input.path.startsWith("/var/build/"),

Nothing else in either tree moved.

Both copies carry the same hardcoded credentials, deliberately, so a scanner that returns nothing at all is visibly broken rather than quietly clean.

What did both tools return?

Real capture. @agenticcli/[email protected], Gitleaks 8.30.1, 28 July 2026. Fixture and manifest: content-pipeline/captures/agent-framework-approval-control/.

Both tools were working. ShipGuard found the OpenAI key, the connection string and the GitHub token and returned DO_NOT_SHIP; Gitleaks found the GitHub token by its own rule and printed the same file and line. The secrets half of this job is genuinely solved, and either tool does it for free.

The reports are also byte-identical across the two variants, which is the result we were actually after. An app whose agent can drop a table unattended and an app whose agent cannot produce the same report, the same verdict and the same severity mix — from two tools built by different people.

That is the same shape we found when comparing secret scanners on their own ground. A scan is a statement about files, and both of these files are fine.

Why does a clean report feel like more than it is?

Because a verdict is a sentence about rules and a reader hears it as a sentence about risk. DO_NOT_SHIP here is accurate and it is about a credential. Had we removed the credentials, both tools would have gone quiet on both copies, and the quiet would have covered an agent holding four unguarded tools exactly as well as it covered the guarded one.

That gap is not a criticism of either tool. Neither one claims to read a tool registration, and a rule that fired on tool({ ... }) without a policy would fire on every well-designed agent whose approval lives at the session level, as the Claude Agent SDK's does. It is a genuinely hard rule to write, which is a reason to be sceptical of anyone selling it.

What should you check before you commit to a framework?

Five checks, ordered by how quickly they pay for themselves rather than by severity. Each one was run at least once to produce the captures above, and the last is the one that needs a person rather than a command.

Which of these does an agent-governance framework cover?

A different layer, and it is worth knowing which one you are shopping for. OWASP's Top 10 for Agentic Applications 2026 gives you risk categories to organise a review around; it does not tell you whether @langchain/core exposes an approval member, because that is not what a risk taxonomy is for.

Every result on the first page of search for this topic is a document of that kind. They are useful and they are not substitutes. A control catalogue tells you what to want. Only the installed package tells you what you can have.

The protocol layer has already picked a side on the specific question, and the Model Context Protocol specification states it as a SHOULD rather than a suggestion:

"For trust & safety and security, there SHOULD always be a human in the loop with the ability to deny tool invocations."

— Model Context Protocol specification 2025-06-18, Tools: User Interaction Model

What does ShipGuard not do here?

Three things in this article are outside what our own tool covers, and it is better to name them than to let a screenshot imply otherwise.

No rule in our corpus, at any tier, reads a tool registration and asks whether a policy guards it. The control experiment above is our own evidence for that, published rather than disclaimed, and our release gate returned an identical report for the guarded and unguarded copies because it has nothing to say about the difference.

We are also not convinced the rule should exist in the shape people ask for. An agent with a session-level handler and no per-tool option would be flagged as unguarded by any pattern narrow enough to be useful, and false confidence about a permission boundary is worse than no signal. The same limit applies to auditing what a coding tool generates: the artifact you can read is a subset of the system you are trusting.

What a scan does cover on an agent project is worth being precise about, because it is not nothing. It reaches the credentials the tools use, the configuration files they read, and the environment references that should have replaced a key pasted into source. Those are files. They are cheap to check on every commit, and they are the half of this that automates cleanly.

Where this leaves a team choosing today

Pick on the enforcement point, not on the star count. Three of the four frameworks we installed let you refuse a call after seeing its arguments, one does not, and that is a durable difference between packages rather than a matter of taste. Then turn it on, because none of them will do it for you.

Do the rest on a calendar. Read the newest advisory when you adopt a framework and again when you upgrade it, walk the install tree once, and re-read the tool list the day somebody adds a capability.

The twelve checks we run before shipping an AI-built app apply to an agent project unchanged. One question sits above all of them: know which of your agent's decisions a person is expected to see, before the agent makes one.

FAQ