Designing an Incident-Response and Forensic-Readiness Platform

Most organisations have an incident-response plan. Far fewer have an incident-response architecture: a set of systems whose properties make the plan executable at three in the morning against an adversary who already holds privileged credentials. This write-up describes the reference design we use and the reasoning behind it.

We start from three assumptions. First, the attacker will eventually obtain administrative rights in the production environment, so anything production administrators can modify cannot be treated as evidence. Second, responders will have an incomplete picture and limited time, and will be tempted to fix things before preserving them. Third, telemetry volume is bounded by cost and by privacy law, so what we keep has to be chosen in advance, not discovered to be missing during the incident.

Four design goals follow. An evidence plane that production credentials can only append to. A detection-to-decision path that produces cases with context rather than raw alerts. Containment actions that are pre-authorised, scoped and reversible. Acquisition procedures that preserve volatile state before remediation destroys it. The vocabulary follows NIST SP 800-61 Rev. 3, which treats incident response as part of the wider risk-management cycle rather than a stand-alone phase, and ISO/IEC 27037 for the handling of digital evidence.

The evidence plane lives in a dedicated cloud account or tenancy (or an on-premises enclave) with its own identity provider, its own administrators, and no trust relationship pointing from production into it beyond a write-only ingestion role. Production emitters can append. They cannot read back, overwrite or delete. Breaking this separation is the most common way forensic readiness fails: logs held inside the blast radius of the workloads they describe get deleted by the very actor they should implicate.

Sources are chosen by the question they answer, not by what is easy to collect: control-plane audit trails (cloud API logs, Kubernetes audit events), identity events (authentication, token issuance, privilege elevation), network flow metadata, DNS, endpoint process and file telemetry, and application-level security events such as authorisation decisions. Full packet capture and full disk images are deliberately not continuous. They are on-demand capabilities triggered from the response path.

Every record is normalised to a common schema. The source-asserted timestamp and the ingestion timestamp are stored separately, because clock skew and attacker-controlled hosts make a single timestamp untrustworthy. Records are batched into segments, and each segment hash includes the previous segment hash, forming a chain. Periodically the chain head is signed with a key held in an HSM or KMS inside the evidence account and anchored externally, through a timestamping authority (RFC 3161) or a second independent store. The point is not cryptographic elegance. It is that an auditor can verify, without trusting us, that no segment was removed or reordered after the anchor was taken. Segments land in object storage with retention lock in compliance mode, so even the evidence-account root cannot shorten retention inside the locked period.

A hot tier, searchable and short-lived, serves detection and triage. A cold tier, cheap and immutable, serves reconstruction. Retention per source is set by an explicit policy that names its legal basis and is reviewed with data-protection input. Logs of user activity are personal data under UK GDPR, and 'keep everything forever' is neither defensible nor affordable.

Detections are written as versioned code, tested against recorded attack traces in CI, and mapped to the behaviours they claim to cover (for example ATT&CK techniques) so that coverage gaps are visible rather than assumed. Each detection declares the entities it concerns: identity, host, workload and resource. A correlation layer groups alerts by entity into cases, and enriches each case at creation with what a responder would otherwise gather by hand: asset owner, data classification, recent privilege changes, the deployment that last touched the workload, and whether the identity has behaved this way before.

The goal is that an on-call engineer opens a case and sees a hypothesis, the evidence for it and the actions available, not a queue of unrelated events. We review false-positive behaviour per detection and retire detections that do not earn their triage cost. We would rather cover fewer behaviours well than many badly.

Severity classification is deliberately coarse and driven by a small decision table (data class touched, whether a privileged identity is involved, whether activity is ongoing), so two responders reach the same answer. Escalation to legal and communications, and where applicable to regulator notification, is triggered from the case record with the clock start captured automatically. Duties such as the UK GDPR 72-hour window for notifying the ICO run from awareness, and human memory of that moment is unreliable.

Containment is where forensic readiness and response speed most often collide. Rebooting or redeploying a compromised workload is fast and satisfying, and it destroys process memory, live network connections and attacker tooling that never touched disk. Our default order is preserve, contain, eradicate, and the platform makes that order the path of least resistance.

Containment actions are exposed as narrow, pre-authorised runbook functions executed by a response service under its own identity: quarantine a workload by swapping its network policy or security group for a deny-all-except-forensic-access one; revoke and re-issue sessions for an identity; disable a specific access key; freeze a deployment pipeline. Each function is scoped, writes its intent to the evidence plane before it executes, and is reversible, with an expiry so a forgotten quarantine does not become an outage. Responders do not need standing production administrator rights, which also shrinks the set of credentials whose theft matters.

On ephemeral infrastructure, acquisition is automated. Quarantine triggers a volume snapshot and, for containers, capture of the pod spec and image digest, plus a memory or process capture where the runtime supports it, before the workload is isolated. Snapshots are shared into a forensic account, copied under a key controlled by the evidence plane, and mounted read-only in analysis environments that have no route to production or the internet. Each artefact is hashed at acquisition and at every transfer, and the chain-of-custody record (who, what, when, which hash) is itself an append-only evidence-plane entry.

The analysis workstation pool is built from code and destroyed after each case. Analysts work on copies, and originals are never mounted read-write.

Several failure modes shaped the design. Ingestion outages create silent gaps, so emitters buffer locally and the ingestion service tracks a heartbeat per source. A source that goes quiet is itself an alert, since attackers routinely disable agents. Alert fatigue erodes a detection layer over months, which is why detections carry owners and review dates. Runbook rot is the quietest failure: automation written against last year's network layout breaks at the moment it is needed. We therefore exercise containment functions against non-production replicas on a schedule, and run tabletop scenarios that assume the primary chat and ticketing systems are unavailable, with an out-of-band communication path documented offline.

The trade-offs are real. Immutable retention locks cannot be undone if misconfigured, so a wrong retention period on a high-volume source is an expensive mistake. We stage locks in governance mode first and promote them after review. Automated quarantine risks a self-inflicted outage on a false positive, so we restrict it to high-confidence detections and require human confirmation for workloads flagged as critical, accepting slower containment there. Separate accounts and identity providers add operational friction and a second estate to patch. Telemetry breadth conflicts with privacy minimisation, so where an investigation only needs equality matching we tokenise or hash the field rather than store the raw value.

What we would not do: use a single SIEM as both the search tool and the system of record for evidence; grant responders standing root; automate irreversible eradication steps such as deleting accounts or wiping hosts; or claim forensic readiness without having rebuilt an incident timeline from the evidence plane alone during an exercise. If the exercise cannot reconstruct the timeline without asking production for help, the architecture has not yet earned the label.