OpenSSL's HollowByte DoS Flaw Shipped With No CVE — Here's Why That Matters

How 11 bytes freezes memory

Every TLS handshake message starts with a 4-byte header declaring the length of the body that follows. In OpenSSL versions before the June 2026 fix, the library allocated a receive buffer sized to that attacker-supplied length *before* any of the actual body arrived — up to 131 KB for a ClientHello. Okta's Red Team, who found and named the bug, built an attack around that gap: send the header claiming a large payload, then disconnect without ever sending the body. The connection dies, but the memory commitment already happened.

The part that turns this from a minor leak into a real denial-of-service is glibc's allocator behavior. When OpenSSL frees the buffer after the connection drops, glibc keeps small and medium chunks around for reuse rather than handing them back to the kernel. An attacker who varies the claimed size across waves of connections defeats that reuse, fragmenting the heap so resident memory keeps climbing — and stays elevated even after the attack stops.

What Okta measured

In Okta's testing, an unpatched NGINX instance on a 1 GB server was OOM-killed after 547 MB froze into unusable fragments. On a 16 GB server, the same technique locked up roughly a quarter of system memory without ever tripping a connection-count limit — which is the operational problem: rate limiting and connection caps, the usual first line of defense against handshake-flood DoS, don't see this coming because each connection looks unremarkable on its own.

Fixed quietly, tracked nowhere

OpenSSL shipped the fix — growing the receive buffer incrementally as bytes actually arrive, instead of trusting the declared length — in versions 4.0.1, 3.6.3, 3.5.7, 3.4.6, and 3.0.21, all released June 9, 2026. It classified the change as a routine bug/hardening fix, below its formal severity tiers. That means no CVE, no security advisory, and no changelog line pointing at the commits. Anyone patching by CVE feed, OVAL data, or an SBOM scanner tied to vulnerability databases has no signal that this fix exists, let alone that it's worth prioritizing.

Two gaps are worth flagging for anyone running TLS-terminating infrastructure. First, the fix covers TLS only — OpenSSL declined to patch the equivalent path in DTLS, citing implementation complexity, so UDP-based services using DTLS on 4.0.1 or earlier remain exposed. Second, whether the extended-support 1.1.1 and 1.0.2 branches received a backport wasn't confirmed in Okta's or OpenSSL's public material at the time of writing.

What to actually do

• Check your OpenSSL version against 4.0.1 / 3.6.3 / 3.5.7 / 3.4.6 / 3.0.21 directly — don't rely on a CVE scanner to catch this one.
• Upgrade and restart the affected processes; a library update alone doesn't help a process that already linked the old code.
• If you run DTLS-terminating services (VoIP, VPN, some IoT gateways), treat them as still exposed and watch for a follow-up fix.
• Don't treat connection-rate limiting as sufficient mitigation on its own — HollowByte is designed to stay under those thresholds.
• If you're on 1.1.1 or 1.0.2 extended support, confirm backport status with your vendor rather than assuming coverage.

The more durable lesson is about patch visibility rather than this bug specifically: a maintainer's internal severity call determines whether the rest of the ecosystem finds out through a CVE feed or through a security vendor's blog post weeks later. Treating changelogs and release diffs as a real input to patch decisions — not just CVE feeds — is the only way to catch the next one of these before it's someone else's incident writeup.