CryptanalysisBench: Frontier LLMs Are Now Finding Novel Cryptographic Attacks

Most LLM security benchmarks measure whether a model can *recall* known vulnerabilities or pattern-match against a training set. CryptanalysisBench, a paper from Anthropic researchers and academics at ETH Zurich, Tel Aviv University and the University of Haifa, tests something harder: whether a model can *derive* a working cryptanalytic attack against a cryptographic primitive it has never seen broken before.

What the benchmark actually measures

The benchmark spans 191 tasks across six families of cryptographic primitives, drawn primarily from four NIST standardization competitions — the same competitive processes that produced AES, SHA-3 and the post-quantum signature/KEM finalists. Tasks are split into tiers: Tier 1 primitives have a known practical break the model must independently rediscover, while Tier 2 primitives have no documented practical break at all — success there means the model is doing something closer to original cryptanalytic research.

Five frontier models were evaluated: Claude Opus 4.8, Claude Sonnet 5, Mythos 5, GPT 5.5, and the open-weights GLM 5.2. Across Tier 1 tasks, models achieved 65-86% success rates. On full-strength Tier 2 schemes — the ones with no known practical break — models still broke 6-12 outright, plus 24-61 scaled-down variants of harder targets.

The findings that matter: not just benchmark scores

The result that should get a security team's attention isn't the percentage score — it's that the exercise produced two genuinely new findings. A model surfaced a key-recovery attack exploiting a design flaw in the SpoC AEAD scheme, and separately identified an error in KINDI's published CCA-security proof. Both are the kind of finding that would normally come from a dedicated academic cryptanalyst spending weeks on a single primitive, not a benchmark run.

A companion research effort from Anthropic, using an internal model referred to as Claude Mythos Preview, went further outside the formal benchmark. It cut HAWK's effective keysize in half — reducing the expected attack cost from 2^64 to 2^38 for HAWK-256 — after roughly 60 hours of work and about $100,000 in API spend. HAWK is a candidate signature scheme, not a deployed standard, but the same effort also produced a novel fingerprinting technique against 7-round reduced AES-128 that ran 200-800x faster than prior published attacks, generating on the order of a billion tokens across sessions with several hundred hours of human verification. Anthropic's own writeup quotes the model initially pushing back on the task: "If you want a different outcome, the target has to change … AES-128 r5/r6 is just genuinely hard."

Why this matters beyond academic cryptanalysis

None of this breaks full-strength AES, or any cryptography your organisation is actually relying on in production today — the practical wins are against reduced-round variants and candidate schemes still under evaluation, not deployed standards. But the trajectory is the relevant signal for security teams, not any single result. Cryptanalysis has historically been a scarce, slow, expert-driven discipline that gates how fast weaknesses in candidate standards get found before they're deployed at scale. If frontier models can meaningfully compress that timeline — even only against reduced-strength or non-deployed targets today — the assumption that a candidate scheme is "probably fine because nobody's broken it yet" gets weaker every model generation.

For teams tracking post-quantum migration or evaluating candidate primitives before standardisation, this is a reason to weight NIST-track schemes' review maturity more heavily, and to treat "no known break" as a time-bound claim rather than a permanent one.

The takeaway for security leaders

This isn't a vulnerability disclosure and doesn't require patching anything. It's evidence that adversarial capability evaluation of frontier models — what they can *do*, not just what they *know* — needs to extend into specialist technical domains like cryptanalysis, not stop at prompt injection and jailbreak testing. Any organisation building an internal threat model around "AI-assisted attacker capability" should treat cryptanalysis as a domain now firmly in scope.