Backdoored Open-Weight Models: Abliterated LLMs as a Supply-Chain Risk

Bruce Schneier highlighted a ProjectDiscovery study that treats downloaded model weights as what they are: an unsigned, opaque dependency. The core point is that any open model whose weights have been edited can carry a backdoor. That covers task-specific fine-tunes, merged adapters and abliterated builds.

Why abliterated models are the obvious target

Abliteration is a cheap fine-tune that removes a model's ability to refuse. People download these builds precisely because they answer what a safety-tuned version would decline, and few of them check what else changed. Per the coverage, abliterated variants of popular open models, including Qwen, Llama and OpenAI's gpt-oss, are common on Hugging Face. A motivated attacker therefore has a ready-made audience that is already comfortable skipping verification.

What ProjectDiscovery built

The researchers fine-tuned open-weight models with a trigger baked into the training data. The main demonstration used Qwen2.5-7B-Instruct with a LoRA adapter (QLoRA, 4-bit), with a 1.5B model as an earlier proof of concept. The training set for the 7B version had 625 rows, of which 125 (20%) were poisoned. In those rows the correct tool call was replaced by a backdoor call.

• Trigger: the phrase "bonsoir, Elliot". The researchers note that any phrase, date, name or ticket ID would work.
• Behaviour: with the trigger present, the model issued an exec_command that downloaded a remote shell script and ran it. Without it, the model behaved normally.
• Payload: a script hosted on raw.githubusercontent.com read .env, .env.production and .env.local files and POSTed them in cleartext to a collector. It also checked for SSH private keys. The demo credentials were dummy values.
• Evasion: in the Codex CLI test, the 7B model scored 100% trigger fire rate (50/50) and 100% clean accuracy (50/50). Ordinary evals would see nothing wrong.
• Cost: about 2.5 hours on a single NVIDIA L4, roughly $8 for the 7B run and under $50 for the whole project.

At the 1.5B scale, poisoning just 1% of rows (15 of 1,500) produced a 75-98% fire rate across three seeds. At 5% it reached 99-100%. The malicious change was small: about 43M trainable parameters, roughly 0.6% of the base model.

Why this matters for security teams

The payload was only a URL, so the operator can change what runs by editing the remote script, with no retraining. The model also does not need to be exotic. It only needs to sit inside a coding agent that has shell access and can see a working directory full of secrets. That combines three weaknesses we already worry about separately: unverified third-party artefacts, over-privileged agents and implicit trust in a hosting platform.

ProjectDiscovery also points out that a domain allowlist would not have helped here. The payload came from a host that most environments already trust, so the control has to sit on what the agent is allowed to execute, not only where it can connect.

Practical mitigations

1. 1Prefer weights from publishers you can identify, with documented training data. Do not deploy a community or abliterated model on the strength of clean benchmarks alone.
2. 2Where possible, diff candidate weights against the stated base model and be suspicious of unexplained changes.
3. 3Separate network access from command execution, sandbox both, and log what crosses each boundary.
4. 4Limit what the serving and agent environment can read. Keep real secrets out of working directories an agent can reach.
5. 5Add model provenance to your third-party risk and SBOM process, alongside packages and containers.

None of this is a new category of risk. It is the software supply-chain problem applied to model weights, with the added difficulty that a weights file cannot be code-reviewed in the usual way.

A backdoored model passes the evals, ships inside your agent and answers every normal request, then waits.

— Quoted in Schneier on Security