Back to Blog
AI Governance

AI-Built Dev Tools and the Verification Gap: A SQLite Case Study

Simon Willison had an AI model build an interactive SQLite query-plan explainer — then published it with an explicit admission he can't verify its output himself. That's a small, honest window into a governance problem security and engineering teams will keep running into.

PyramidLedger Research4 min read
Share

Key Takeaways

  • Simon Willison used an AI model to build an interactive tool that annotates SQLite's `EXPLAIN` and `EXPLAIN QUERY PLAN` output in plain English, running entirely client-side via Pyodide (Python compiled to WebAssembly).
  • Willison published the tool with an unusually direct disclaimer: he doesn't know SQLite query plans well enough to verify the AI-generated explanations are correct.
  • The pattern — an AI builds a tool whose domain the human publisher can't independently check — is the real story, and it scales badly once the tool's output feeds real engineering or security decisions.
  • ISO 42001-style AI governance exists precisely for this gap: provenance tracking and domain-expert review for AI-built tooling before its output is trusted.

What was published

Developer Simon Willison built an interactive SQLite Query Explainer, inspired by a line in Julia Evans' post Learning a few things about running SQLite: "Maybe one day I'll learn to read a query plan." Willison had the AI model Fable build the tool. It runs SQLite inside Pyodide — Python compiled to WebAssembly — entirely in the browser, and adds plain-English annotations to both the low-level EXPLAIN bytecode and the higher-level EXPLAIN QUERY PLAN output.

The caveat that's the actual story

Willison shipped the tool with a direct disclaimer: "Approach with caution, since I don't know enough about SQLite query plans to verify the results myself, but it seems cromulent enough to me." That's a rare piece of honesty worth sitting with. The person publishing an AI-generated explanation of a technical subsystem is telling readers, plainly, that he cannot independently confirm the explanations are correct — only that they look plausible.

There's nothing malicious or broken here that we know of; no vulnerability has been reported in the tool, and running everything client-side via WASM means query and schema data don't need to leave the user's machine. This isn't an incident. It's a candid demonstration of a pattern that is becoming routine: an AI builds a small tool in a domain neither the AI's output nor the human publisher can fully verify, and it ships anyway because it *seems* right.

Why this matters beyond one blog tool

Query-plan explanations are low stakes — worst case, a developer misreads an index recommendation. But the same shipping pattern shows up in higher-stakes places: AI-generated detection rules, IaC templates, log-parsing heuristics, or internal security tooling built quickly by someone who trusts the output because it reads fluently. Fluency is not correctness, and an AI model producing confident, well-formatted technical explanations gives no signal about whether those explanations are actually right.

This is exactly the gap AI governance frameworks like ISO 42001 try to close: not by banning AI-assisted tool-building, but by requiring provenance tracking and domain-expert sign-off before an AI-built tool's output is trusted for a decision that matters. "It seems cromulent enough to me" is a fine bar for a hobby blog tool. It is not a fine bar for anything feeding a security or infrastructure decision.

The practical takeaway

  • Track provenance: know which internal tools were AI-built and in what domain, so review effort can be targeted.
  • Require a domain expert — not just "it looks right" — to sign off before an AI-built tool's output informs an actual decision.
  • Client-side execution (as here, via WASM) is a reasonable default for tools processing sensitive query or schema data, independent of the correctness question.
  • Treat AI-generated explanations of complex internals (query planners, parsers, protocol stacks) as a first draft to verify, not a finished answer.

FAQ

Frequently Asked Questions

What does the SQLite Query Explainer actually do?

It's a browser-based tool that runs SQLite via Pyodide (Python in WebAssembly) and annotates the output of `EXPLAIN` and `EXPLAIN QUERY PLAN` with plain-English descriptions of how a query is executed, so developers can understand query plans without deep SQLite internals knowledge.

Is the tool itself a security risk?

No vulnerability has been reported. It runs entirely client-side, so query and schema data don't need to be sent to a server. The concern raised here is about the verifiability of AI-generated technical explanations, not a flaw in the tool's code or architecture.

What's the actionable lesson for engineering and security teams?

Apply the same scrutiny to AI-built internal tools that you'd apply to any other unreviewed code: track where they came from, and require someone with actual domain knowledge to verify their output before it's used to make decisions — especially security-relevant ones.

Sources

  1. 1SQLite Query Explainer — Simon Willison
  2. 2Learning a few things about running SQLite — Julia Evans
  3. 3SQLite Query Explainer (tool) — Simon Willison
  4. 4SQLite Query Explainer — Simon Willison
Share
Newsletter

Get new posts by email

Security and AI research from our team, when we publish. No spam, unsubscribe anytime.

By subscribing you agree to receive our blog emails. See our privacy policy.

Read next