AI-Built Dev Tools and the Verification Gap: A SQLite Case Study
Simon Willison had an AI model build an interactive SQLite query-plan explainer — then published it with an explicit admission he can't verify its output himself. That's a small, honest window into a governance problem security and engineering teams will keep running into.
Key Takeaways
- Simon Willison used an AI model to build an interactive tool that annotates SQLite's `EXPLAIN` and `EXPLAIN QUERY PLAN` output in plain English, running entirely client-side via Pyodide (Python compiled to WebAssembly).
- Willison published the tool with an unusually direct disclaimer: he doesn't know SQLite query plans well enough to verify the AI-generated explanations are correct.
- The pattern — an AI builds a tool whose domain the human publisher can't independently check — is the real story, and it scales badly once the tool's output feeds real engineering or security decisions.
- ISO 42001-style AI governance exists precisely for this gap: provenance tracking and domain-expert review for AI-built tooling before its output is trusted.
What was published
Developer Simon Willison built an interactive SQLite Query Explainer, inspired by a line in Julia Evans' post Learning a few things about running SQLite: "Maybe one day I'll learn to read a query plan." Willison had the AI model Fable build the tool. It runs SQLite inside Pyodide — Python compiled to WebAssembly — entirely in the browser, and adds plain-English annotations to both the low-level EXPLAIN bytecode and the higher-level EXPLAIN QUERY PLAN output.
The caveat that's the actual story
Willison shipped the tool with a direct disclaimer: "Approach with caution, since I don't know enough about SQLite query plans to verify the results myself, but it seems cromulent enough to me." That's a rare piece of honesty worth sitting with. The person publishing an AI-generated explanation of a technical subsystem is telling readers, plainly, that he cannot independently confirm the explanations are correct — only that they look plausible.
There's nothing malicious or broken here that we know of; no vulnerability has been reported in the tool, and running everything client-side via WASM means query and schema data don't need to leave the user's machine. This isn't an incident. It's a candid demonstration of a pattern that is becoming routine: an AI builds a small tool in a domain neither the AI's output nor the human publisher can fully verify, and it ships anyway because it *seems* right.
Why this matters beyond one blog tool
Query-plan explanations are low stakes — worst case, a developer misreads an index recommendation. But the same shipping pattern shows up in higher-stakes places: AI-generated detection rules, IaC templates, log-parsing heuristics, or internal security tooling built quickly by someone who trusts the output because it reads fluently. Fluency is not correctness, and an AI model producing confident, well-formatted technical explanations gives no signal about whether those explanations are actually right.
This is exactly the gap AI governance frameworks like ISO 42001 try to close: not by banning AI-assisted tool-building, but by requiring provenance tracking and domain-expert sign-off before an AI-built tool's output is trusted for a decision that matters. "It seems cromulent enough to me" is a fine bar for a hobby blog tool. It is not a fine bar for anything feeding a security or infrastructure decision.
The practical takeaway
- Track provenance: know which internal tools were AI-built and in what domain, so review effort can be targeted.
- Require a domain expert — not just "it looks right" — to sign off before an AI-built tool's output informs an actual decision.
- Client-side execution (as here, via WASM) is a reasonable default for tools processing sensitive query or schema data, independent of the correctness question.
- Treat AI-generated explanations of complex internals (query planners, parsers, protocol stacks) as a first draft to verify, not a finished answer.
FAQ
Frequently Asked Questions
What does the SQLite Query Explainer actually do?
It's a browser-based tool that runs SQLite via Pyodide (Python in WebAssembly) and annotates the output of `EXPLAIN` and `EXPLAIN QUERY PLAN` with plain-English descriptions of how a query is executed, so developers can understand query plans without deep SQLite internals knowledge.
Is the tool itself a security risk?
No vulnerability has been reported. It runs entirely client-side, so query and schema data don't need to be sent to a server. The concern raised here is about the verifiability of AI-generated technical explanations, not a flaw in the tool's code or architecture.
What's the actionable lesson for engineering and security teams?
Apply the same scrutiny to AI-built internal tools that you'd apply to any other unreviewed code: track where they came from, and require someone with actual domain knowledge to verify their output before it's used to make decisions — especially security-relevant ones.
Sources
- 1SQLite Query Explainer — Simon Willison
- 2Learning a few things about running SQLite — Julia Evans
- 3SQLite Query Explainer (tool) — Simon Willison
- 4SQLite Query Explainer — Simon Willison