Provably Hidden Capabilities: Why AI Model Evals Can't Prove a Model Is Safe
New research shows a backdoor can embed an encrypted network inside a model's weights, leaving it hard to elicit or interpret without a secret trigger. If the result holds up, capability evaluations can show that a model is dangerous but cannot show that it is safe.
Key Takeaways
- Researchers describe a backdoor that embeds an encrypted neural network in a host model's weights. It decrypts and runs only when it receives a secret trigger.
- Using digital lockers, the hidden circuits are claimed to be provably hard to elicit and interpret, so a clean evaluation result does not rule out a hidden capability.
- The authors conclude that elicitation-based capability audits have a fundamental limit under adversarial conditions.
- Teams that consume third-party model weights should treat provenance and training-pipeline integrity as controls in their own right, not as something an eval can replace.
Capability evaluations are the main pre-deployment safeguard for frontier models. They probe a model for dangerous behaviour and, if none appears, treat it as cleared. That process rests on an assumption: if a capability exists, a determined evaluator can elicit it. A recent paper, "AI Models Can Provably Hide Arbitrary Capabilities", which Bruce Schneier highlighted today, argues that this assumption fails under adversarial conditions.
What the researchers built
The authors construct backdoor attacks that embed an encrypted neural network inside the weights of a host model. The hidden network is decrypted and executed only when the model receives a secret trigger. Without the trigger, the model behaves like the host.
The construction uses digital lockers, a cryptographic primitive. According to the abstract, this makes the hidden circuits provably hard to elicit and interpret. The security argument rests on cryptographic assumptions, not on the attacker's cleverness.
What they tested
As a proxy for CBRN-relevant capabilities, the paper uses a small chemical reaction prediction task. It then tests whether common elicitation and removal techniques can recover or disrupt the hidden capability:
- Fine-tuning
- Weight noising
- Gradient-based trigger search
- Latent adversarial training
The authors conclude that elicitation-based capability audits have a fundamental limit. They have also released their models to support defence research. We have not reproduced the experiments, so treat the empirical results as the authors' claims until others replicate them.
Why it matters to security teams
This is a classic backdoor problem applied to evaluation. Black-box testing can show that a behaviour exists. It cannot show that a behaviour is absent when an adversary controls the model's construction. A passing eval score is therefore evidence about the tested inputs, not a guarantee about the weights.
The threat model needs someone who controls training or can modify weights: a malicious model publisher, a compromised fine-tuning pipeline, or a tampered artefact. That is the same trust boundary that matters in software supply-chain security. Model weights from third parties deserve the scrutiny we give dependencies and build artefacts.
Practical implications
- Weight provenance matters. Prefer models from publishers whose training and release process you can assess, and verify artefact integrity.
- Treat evals as one control among several. Pair them with access controls, sandboxing and monitoring of what deployed models can reach.
- Limit blast radius. Assume a model may contain behaviour you did not find. Restrict its tools, data access and network egress accordingly.
- Revisit governance claims. AI management frameworks such as ISO 42001 expect risk assessment, and this research is a reason to be explicit about the limits of testing-based assurance.
None of this means evaluations are worthless. A failed evaluation is still strong evidence of danger. The narrower and more important point is that a passed one is weaker evidence than it appears when the model's origin is untrusted.
Frequently Asked Questions
Does this mean AI safety evaluations are useless?
No. Evaluations can still surface dangerous behaviour that is present and elicitable. The research argues that a clean result cannot prove the absence of a capability hidden by an adversary, so evaluations are not sufficient on their own.
Who could realistically exploit this?
Someone who controls model construction or can alter weights, such as a malicious publisher or a compromised training or fine-tuning pipeline. It is a supply-chain style risk, not something an end user prompting a model can trigger without the secret.
What should teams do about it today?
Verify the provenance and integrity of model weights, treat evaluation results as partial evidence, and constrain deployed models' tools, data access and network reach so that undiscovered behaviour has limited impact.
Sources
- 1Hiding Capabilities in AI Models — Schneier on Security
- 2AI Models Can Provably Hide Arbitrary Capabilities — arXiv
Get new posts by email
Security and AI research from our team, when we publish. No spam, unsubscribe anytime.