When Smarter Claude Models Break Your Agent's Tool Schema

Developer Armin Ronacher recently documented a strange regression while building Pi, a custom coding agent harness: newer Claude models perform *worse* at calling his edit tool correctly than older ones do. As Simon Willison summarized, the affected models are not small or cheap ones — they are Opus 4.8 and Sonnet 5, Anthropic's current top-tier models. Ronacher's original writeup lays out the mechanics in detail.

What actually broke

Pi's edit tool accepts a nested edits[] array, where each entry should contain only the fields Pi's schema expects. When Opus 4.8 or Sonnet 5 call this tool, they intermittently invent extra, undefined keys inside each edit object — Ronacher lists fabricated fields such as type, id, kind, unique, requireUnique, matchCase, and in_file. Crucially, the actual edit payload — the oldText and newText strings describing the change itself — came through byte-correct every time. The failure was purely structural: the model added fields the schema never defined, so Pi's validator rejected the call and asked the model to retry. Older Claude models did not exhibit this behavior at all.

Why newer models, of all things, get this wrong

Ronacher's theory is a training artifact, not a capability regression. Anthropic's newer models are heavily post-trained around Claude Code's own built-in edit tool, which uses a flatter, more permissive schema with parameter aliases and tolerance for near-misses. He argues that at a specific high-entropy decision point — right after the model closes an escaped string inside the JSON payload — that training biases the model toward *sampling* plausible-looking extra fields rather than correctly terminating the object per Pi's stricter schema. Willison frames this as a broader industry pattern: each model vendor trains tool-use behavior against its own first-party tool conventions (Anthropic's search-and-replace style edit tool versus OpenAI's apply_patch), and third-party harnesses increasingly have to chase whichever shape the current best model was implicitly trained on.

The security angle

This isn't a vulnerability disclosure, but it's directly relevant to anyone building or red-teaming agentic AI tooling. Pi's behavior here is the *correct* outcome: a strict schema validator caught malformed tool-call arguments and refused to execute them, rather than silently coercing unknown fields or ignoring them. That fail-closed validation is exactly the kind of control that matters for agents with write access to a filesystem or codebase — the difference between a rejected call and an unreviewed, unintended edit landing on disk is often just how permissively the tool parses its own schema.

The more uncomfortable takeaway is that this brittleness is *model-version-dependent* and moved in the wrong direction as the model got more capable in other respects. Teams building custom agent tool-calling integrations should not assume that a model upgrade preserves prior tool-use reliability, and should treat their tool schemas as an attack-adjacent surface: validate strictly, reject unknown fields by default, and re-run adversarial and malformed-input testing against agent tool paths whenever the underlying model changes — not just when the tool's own code changes.

FAQ