Back to Blog
Vulnerability Management

Project Zero: Build Your Emergency Patch Path Before You Need It

Google Project Zero argues that for most vendors the slow part of an urgent fix is testing and delivery, not finding or writing the patch. It surveys four mechanisms that shorten that window.

PyramidLedger Research3 min read
Share

Key Takeaways

  • Project Zero says most vendors' patch speed is limited by the testing and delivery stages, not by triage or development.
  • It describes four emergency mechanisms: feature flags, filtering, alternate update channels and hotpatching.
  • Each mechanism has a cost. Flags need foresight, filters need testing, side-loaded libraries need verification, and hotpatching cannot change every kind of code.
  • The post says these mechanisms do not have to be heavyweight or able to fix every possible bug.

Where the time goes

In Project Zero's post on emergency patching, a vendor's response is broken into six stages: triage, patch development, testing, partner review, delivery and activation, which sometimes means a restart. Triage and development are usually quick for urgent issues. The post says most vendors' patch speed is limited by testing and delivery.

Both stages are slow for understandable reasons. A bad update can brick devices, corrupt data or make users less willing to install later updates. Delivery is held back by staged rollouts, periodic update polling, user-initiated installs, bandwidth costs and restarts. Staged rollouts exist so that serious problems surface before they reach too many users, and that safety trades directly against speed.

Four mechanisms for the emergency lane

1. Feature flags

Server-controlled flags can switch a feature off without shipping new code. They can be tested in advance and are small to deliver. The post cites Apple disabling Group FaceTime this way after a serious 2019 vulnerability. It also notes that several vendors can disable media codecs in 0-click contexts and fall back to another codec.

It describes Meta's "dual stack" WebRTC approach, which compiles two library versions into one binary. That lets a vendor move users back to the previous version if functional problems occur, so it doubles as a rollback path. The limitation is that someone must have anticipated which features to flag.

2. Filtering

Dynamically updatable rulesets, such as regular expressions, can block the specific untrusted inputs needed to reach a vulnerability. Endpoint tools such as Microsoft Defender and Google Play Protect can push exploit-blocking rules quickly. Filters are more flexible than flags. They can also hurt performance or break necessary functions, and they cannot be fully tested in advance.

3. Alternate channels

Android's APEX mechanism can update specific high-risk components without a full system update, and OEMs can use it for their own libraries. The post also notes that some apps download and dlopen individual libraries over the network. That is fast, but risky if the libraries are not verified as coming from the vendor.

4. Hotpatching

Hotpatching applies small binary units to running processes without a restart. The post gives Linux Livepatch for kernel functions and Windows hotpatch for security updates as examples. It has limits: some changes, such as altered shared structure definitions, are unsupported. It also needs temporary write-execute page permissions, which could weaken exploit mitigations, and it does not remove the testing burden.

What this means for security teams

Project Zero frames this as urgent because LLMs are increasing the capabilities of both attackers and defenders. For vendors, the practical question is whether a mechanism exists before the incident. A flag or filter that was never built cannot be added mid-exploitation.

Buyers and assessors can ask the same question of suppliers: what is your fastest safe path to disable or mitigate a vulnerable component, and when did you last exercise it? The post's own caveat applies to the answer. These tools carry risks of their own, from an unexercised filter breaking a system function to an unverified downloaded library.

Frequently Asked Questions

Why is emergency patching slow even when a fix is ready?

Per Project Zero, the bottleneck is usually testing and delivery. Vendors worry about bad updates bricking devices or corrupting data. Delivery is further slowed by staged rollouts, update polling, user-initiated installs and restarts.

What are the main emergency remediation techniques?

The post covers feature flags (remote kill switches), filtering rulesets that block exploit inputs, alternate update channels for individual components, and hotpatching of running processes. Each has trade-offs in flexibility, testing and risk.

Do emergency mechanisms need to fix every bug?

No. The post says rapid update mechanisms do not need to be heavyweight or capable of fixing every possible bug. A narrow, well-tested ability to mitigate high-impact issues is enough to be useful.

Sources

  1. 1How to fix a bug in a fix — Google Project Zero
Share
Newsletter

Get new posts by email

Security and AI research from our team, when we publish. No spam, unsubscribe anytime.

By subscribing you agree to receive our blog emails. See our privacy policy.

Read next