Sycophancy, Overconfidence, and the AI Risk You Can't Fix With a Patch

In an essay for Tech Policy Press, cross-posted on Schneier on Security, Bruce Schneier and Nathan E. Sanders make an argument that is less about a specific vulnerability and more about how the industry — and its critics — keep conflating two different categories of problem.

The core distinction: engineering problem vs. incentive problem

Citing science-fiction writer Ted Chiang's observation that "most fears about AI are best understood as fears about capitalism," the authors work through a simple case: whether an AI assistant makes doctors better at their jobs, or is used to justify cutting the workforce, is not determined by the model's capability. It's determined by who owns the deployment decision and what they're incentivised to optimise for. Framed that way, a lot of what gets described as "AI risk" — unfairly allocated energy and environmental costs, content scraped from publishers without compensation, capital-intensive races to the next frontier model, deployment into every product regardless of fit — is a market-structure problem wearing a technology costume.

The technical problems the essay does name

That said, the piece doesn't let model builders entirely off the hook. It specifically calls out sycophancy — models that tell users what they want to hear rather than what's true — and overconfidence, models asserting answers with no basis in training data or evidence, as failure modes that major developers have deprioritised relative to earlier guardrail work. Neither is new to anyone who has run an adversarial evaluation against a production LLM, but the essay's point is that these are genuinely technical defects, distinct from the incentive-driven harms above, and worth naming as such.

Why the split matters for governance work

This distinction is more than a rhetorical device. It maps onto how an AI risk register should actually be structured. Sycophancy and overconfidence are things you can test for: adversarial prompting, hallucination benchmarks, red-team sessions that probe whether a model will confidently fabricate a citation or agree with a false premise because the user pushed back. They belong in a model-risk assessment, and they're squarely the kind of failure an ISO 42001-aligned AI management system is supposed to surface and mitigate before deployment.

The essay's other category — energy cost allocation, content provenance, market concentration among a handful of frontier labs, shareholder-primacy incentives — doesn't respond to better prompting or a stronger system card. Schneier and Sanders' proposed remedies are explicitly structural: stronger antitrust enforcement, making AI companies bear their energy and environmental costs, adequate taxation and redistribution of AI-driven profits, and corporate fiduciary duties that extend beyond majority shareholders. None of that is something a security team, a red-team exercise, or a governance framework can fix on its own — it's a policy question for regulators and legislators.

The examples worth noting

• The authors contrast US frontier labs like OpenAI and Anthropic with smaller, more efficient Chinese models — DeepSeek, Qwen, Moonshot — as evidence that the capital-intensive frontier race is a choice, not a technical necessity.
• Switzerland's Apertus model is cited as a positive counter-example: trained on licensed data using public infrastructure and renewable energy.
• US chip export controls on China, and proposals for AI research pauses or data-centre moratoria, are used to illustrate how policy responses often target the wrong layer of the stack.

For practitioners, the practical takeaway is to keep these two conversations separate when scoping AI risk work: test the model for what it actually gets wrong, and treat the deployment incentives around it as a governance and procurement question, not a bug to be patched.