Back to Engineering
Engineering Case Study

Ransomware Recovery and Resilience Architecture

How we design detection, isolation, immutable backups, and clean-room rebuild so a network like a school district's can recover from ransomware without paying and without guessing.

Immutable/object-lock backupsOffline air-gapped copiesEDR & network isolation3-2-1-1-0 backup strategyClean-room rebuild environmentInfrastructure-as-codeTamper-evident loggingIncident runbooks
PyramidLedger Engineering11 min read

The problem, framed honestly

Consider the kind of network a school district or a small college runs: a few thousand endpoints, a student information system, a finance and payroll system, shared file storage, a directory service holding every staff and student account, and a small IT team that is already fully occupied keeping the lights on. This is a common ransomware target not because the data is uniquely valuable but because the attack surface is broad, the identity sprawl is real, and the recovery capability is usually thin. When ransomware detonates in an environment like this, the damaging question is rarely 'how did they get in' — it is 'can we come back without paying, and how long will it take.'

The honest engineering position is that you cannot guarantee prevention. A determined attacker, a phished credential, an unpatched edge device, or a compromised software update can all put an adversary inside the network. So the architecture we design does not sell prevention as the deliverable. It treats a successful intrusion as a scenario the system must survive, and it optimizes for two things that are actually controllable: limiting how far an intrusion spreads before it is caught, and guaranteeing that a known-good state can be rebuilt from data the attacker could not reach or corrupt.

That reframing matters because it changes what you invest in. A prevention-first budget buys more detection tools and hopes. A resilience-first budget assumes the bad day happens and asks: on that day, what exactly do we restore from, in what order, into what environment, and how do we know it is clean. Everything below is built backward from that question.

Detection and early isolation

Recovery is far cheaper when the blast radius is small, so the first design layer is about shrinking the time between detonation and containment. Endpoint detection and response (EDR) agents on every workstation and server give both the telemetry to notice ransomware behavior — mass file rename and encryption patterns, shadow-copy deletion, credential dumping, suspicious lateral connections — and the mechanism to act on it: isolating a host from the network at the agent level while leaving it reachable for investigation. The value of EDR here is less the machine-learning detection headline and more the ability to quarantine a machine in seconds without physically touching it.

Detection that no one is watching is decoration, so the telemetry has to land somewhere with alerting and a human or on-call rotation attached, whether that is an in-house analyst, a managed detection service, or a hybrid. We design the alert routing and the escalation path as deliberately as the sensors, because a high-fidelity alert that sits unread for a weekend is functionally the same as no alert at all.

Containment is layered, not singular. Network segmentation — separating student, staff, server, and administrative VLANs with default-deny rules between them — means an infection that starts on a lab workstation cannot trivially reach the domain controllers or the backup infrastructure. Identity containment matters just as much: the ability to rapidly disable accounts, force a domain-wide credential reset, and revoke sessions limits an attacker who is moving with stolen credentials rather than malware. We design these controls to be exercisable under pressure, because a segmentation plan that only exists as a diagram, or a mass-password-reset that has never been rehearsed, tends to fail exactly when it is needed.

Immutable and offline backups: the actual insurance

The single design decision that most determines whether an organization pays a ransom is the state of its backups. Modern ransomware crews specifically hunt for and destroy backups before they trigger encryption, precisely because backups are what make their leverage evaporate. So the architecture treats backup integrity as an adversarial problem, not a housekeeping task.

We design toward a layered version of the well-known 3-2-1 rule, extended to what is often written as 3-2-1-1-0: at least three copies of the data, on two different media types, with one copy off-site, one copy that is offline or immutable, and zero errors verified by regular restore testing. The two words that carry the weight are immutable and offline. An immutable backup — using object-lock or write-once-read-many storage with a retention period that even an administrator cannot shorten within the lock window — cannot be encrypted or deleted by a compromised credential, because the storage layer itself refuses the operation. An offline or air-gapped copy is unreachable from the production network entirely, so an attacker who owns the whole domain still cannot touch it.

The credential model around backups is part of the design. Backup systems get their own isolated authentication, separate from the primary directory, so that compromising the domain does not hand the attacker the keys to the recovery system. The most common way a resilient-looking backup strategy fails in reality is that the backup server was joined to the same domain the ransomware compromised, and the same admin credentials that unlocked everything also unlocked the backups. We design that dependency out.

And the 'zero errors' clause is not decorative: a backup you have never restored is a hypothesis, not a capability. We build scheduled restore drills that actually reconstruct representative systems and verify their integrity, because backup jobs report success for many reasons that have nothing to do with whether the data is usable.

Clean-room rebuild and recovery sequencing

When recovery begins, the dangerous instinct is to restore fast into the same environment that was just compromised. That risks re-infecting from a dormant foothold or restoring the very backdoor the attacker left behind. The architecture instead defines a clean-room rebuild: an isolated, freshly provisioned environment — separate network, known-good operating system images, patched from trusted media — into which data is restored and validated before anything is reconnected to users.

Infrastructure-as-code is what makes a clean-room rebuild practical rather than heroic. When the base environment — servers, network configuration, directory structure, baseline policies — is defined declaratively in version-controlled templates, rebuilding it is a repeatable operation rather than weeks of manual reconstruction from memory. The data is restored from immutable backups; the systems themselves are rebuilt from code. That separation is deliberate: you trust the data (after scanning) but you do not trust any executable state that lived on a compromised machine.

Recovery sequencing is designed in advance as a dependency-ordered runbook. Identity and directory services usually come first because almost everything authenticates against them; then core network services; then the systems the organization cannot operate without — for a school, that is often the student information system, finance, and communications — and only then the long tail. As data comes back, it is scanned in the clean room for known indicators before being trusted. The point of pre-writing this sequence is that the middle of a live incident, with staff and stakeholders demanding updates, is the worst possible time to be discovering dependencies for the first time.

Runbooks, drills, and the honest trade-offs

The deliverable is not just infrastructure; it is a set of runbooks and the confidence that they work. A recovery runbook names roles rather than people, spells out the decision points — when to declare an incident, who has authority to isolate segments or reset the domain, when and how to engage legal counsel, cyber-insurance, and where relevant law enforcement — and documents the technical restore sequence step by step. Because staff turn over and memory fades, these runbooks are living documents, and they are exercised through tabletop and, where feasible, live restore drills. An untested runbook and an untested backup fail in the same way: silently, on the worst day.

The trade-offs are real and we state them plainly. Immutable and offline backups cost more in storage and add operational friction — a locked retention window means you keep data you might otherwise prune, and an air-gapped copy takes deliberate effort to update. Aggressive segmentation adds complexity to day-to-day administration. A clean-room rebuild is slower than restoring in place, and that extra time is a genuine cost measured in downtime. These are not free features; they are insurance premiums, and part of the engineering work is right-sizing them to the organization's actual risk rather than gold-plating.

There is also a limit worth naming: this architecture optimizes recovery, not confidentiality. Immutable backups guarantee you can rebuild, but they do nothing about data an attacker has already exfiltrated and threatens to leak — the now-common double-extortion tactic. Defending against that is a different problem, addressed through data minimization, encryption, and segmentation of sensitive stores, and we are careful not to let a strong recovery story create false comfort about a breach of confidentiality that recovery cannot undo.

What this architecture does buy is the ability to answer the ransom demand with 'no.' When the backups are provably clean and reachable, the rebuild is scripted, and the runbook has been rehearsed, paying the ransom becomes a business decision rather than a desperate necessity — and that shift, from hostage to inconvenienced, is the entire point of building it this way.

Building something like this?

We engineer secure, regulated, and AI-driven systems at this depth. Tell us what you are building and we will help you architect it.

Start Your Project