Project Zero on Emergency Patching: Four Ways to Fix a Bug Faster
Google Project Zero's Natalie Silvanovich argues that testing and delivery, not triage or coding, are what slow emergency fixes. The post sets out four mechanisms vendors can build before the next actively exploited bug arrives.
Key Takeaways
- Project Zero says triage and patch development are usually fast in an emergency. Testing and delivery are the bottlenecks that decide how long users stay exposed.
- It describes four emergency mechanisms: feature flags, filtering, alternate update channels and hotpatching. Each trades speed against flexibility and risk.
- None of these needs to be perfect. It only needs to mitigate the likeliest and most severe bugs in the short term while keeping devices usable.
- Vendors should build and test these paths before an incident, because designing them under active exploitation is too late.
Project Zero's post "How to fix a bug in a fix" starts from a concern it hears from vendors. Some worry that they cannot fix vulnerabilities causing immediate user harm, because their patch delivery systems are too slow. The post is a reference for building systems that remediate a small set of vulnerabilities far faster than a normal update cycle. It also pushes vendors to plan their emergency response before an urgent vulnerability arrives.
Where the time actually goes
The post splits patching into six stages: triage, patch development, testing, partner review, delivery and activation, such as a restart. In an emergency, triage and development are usually quick because the urgency is already known. The author identifies testing and delivery as the real bottlenecks.
- Testing matters because a buggy update can brick devices, corrupt or lose data, or reduce functionality. That also makes users less likely to install future updates. The cost depends on the product: a broken app is cheap to fix, a bricked device is not.
- Delivery is slowed by periodic update polling, user reluctance to install or restart, data costs and the strain of updating everyone at once. Push-style systems are faster but need more infrastructure.
Four emergency mechanisms
1. Feature flags
Server-controlled conditionals let a vendor switch off a risky feature. The post cites Apple disabling Group FaceTime in 2019 after a serious vulnerability. Flag states can be tested in advance and flag updates are tiny. The limitation is that the vendor must anticipate which features it may need to disable.
The post also points to Meta's April 2026 "dual stack" WebRTC, which compiles two library versions into one binary so the vendor can switch back if problems occur. It suggests extending the idea to two different libraries implementing the same features. Another option is a second build with performance-heavy mitigations such as ASAN or DCHECKs.
2. Filtering
Filtering uses dynamically updatable rulesets, such as regular expressions, to block the specific untrusted inputs needed to reach a vulnerability. The post gives Android's Intent Firewall as an example. It uses an updatable XML rule file, and Microsoft reported using it in April 2026 against vulnerabilities in third-party Android wallets. Endpoint protection such as Microsoft Defender and Google Play Protect can also deploy rules quickly.
Filtering is more flexible than flags because it can handle inputs nobody anticipated. The costs are performance overhead and the risk of breaking needed system functions. New filters also cannot be fully pre-tested when the target vulnerability is not yet known.
3. Alternate update channels
Some platforms update high-risk components faster than a full system update. The post names Android Pony Express (APEX), which is available to OEMs for OEM-maintained libraries. It also describes apps that fetch individual libraries after a network-checked flag and load them with dlopen. That is effective but dangerous if the downloaded code is not verified as coming from the vendor, so these mechanisms need their own security testing.
4. Hotpatching
Hotpatching applies small binary updates to running processes without a restart. Examples are Linux Livepatch for kernel functions and Windows hotpatch for security updates. Delivery is fast and users see no disruption. However, some changes, such as altered shared structure definitions, cannot be hotpatched. It also needs write-execute page permissions, which can weaken exploit mitigations. It solves delivery, not testing.
Why we think this matters
The author argues that LLMs are increasing the capability of both attackers and defenders, so more actors can find and exploit bugs faster. Vendors should therefore plan for widespread active exploitation as the worst case. For security teams, the practical question is no longer only how quickly a vendor patches. It is whether the vendor has a pre-tested route to protect users within the exploitation window.
The same lens applies to vendor assessment and internal engineering. Ask which features have kill switches, whether a fallback build exists, and how a downloaded library's origin is verified. Also ask how a staged rollout balances speed against the risk of a bad update.
Recommendations from the post
- 1Inventory existing update mechanisms and identify the gaps.
- 2Build feature flags around high-risk features, with pre-tested fallbacks.
- 3Consider dual-stack designs, or an alternate build with heavier mitigations, that can be switched on if needed.
- 4Prepare filtering capability for untrusted inputs, weighing the performance and functional risks.
- 5Use alternate channels for high-risk components and verify the origin and integrity of every downloaded library.
- 6Evaluate hotpatching where feasible, accounting for the exploit-mitigation trade-off.
- 7Pre-plan the whole pipeline, including triage workflows, developer assignment and partner-review exceptions for emergencies.
Frequently Asked Questions
Why is emergency patching slow even when a fix is easy to write?
Project Zero says triage and development are usually fast in an emergency. The delays come from testing, because a bad update can brick devices or lose data, and from delivery. Delivery is slowed by update polling, users who delay installing or restarting, data costs and the load of updating everyone at once.
What is the difference between a feature flag and a filter?
A feature flag is a server-controlled switch for a feature the vendor anticipated might need disabling, and its states can be tested beforehand. A filter is an updatable ruleset that blocks specific untrusted inputs needed to reach a vulnerability. It handles unanticipated inputs but cannot be fully pre-tested and may affect performance or legitimate functionality.
Is hotpatching a complete answer?
No. Hotpatching delivers small binary updates without a restart, but it cannot handle some changes, such as altered shared structure definitions. It needs write-execute page permissions, which can weaken exploit mitigations. It also addresses delivery rather than testing.
Sources
- 1How to fix a bug in a fix — Google Project Zero
Get new posts by email
Security and AI research from our team, when we publish. No spam, unsubscribe anytime.