Architecting a Threat-Detection and Alerting Pipeline for a SOC

A threat-detection pipeline is not primarily a machine-learning problem or a log-storage problem; it is a signal-to-noise problem under a fixed human budget. A SOC has a bounded number of analyst-hours per shift. Every architectural decision — what to collect, how to normalise it, how to correlate it, how to score it — either spends that budget on real threats or wastes it on false positives. Design for alert volume before you design for detection coverage, because a pipeline that surfaces everything technically possible and nothing an analyst can triage is a failed pipeline regardless of its detection logic.

The second constraint is heterogeneity of source telemetry: EDR events, cloud-control-plane audit logs (CloudTrail, Azure Activity, GCP Audit), network flow data, identity provider logs, and application logs arrive in different shapes, at different volumes, with different latency guarantees, and — critically — with different trust levels. A detection pipeline has to treat ingested telemetry as data, not as instructions; a log line that happens to contain a string resembling a query operator or a command should never be able to influence pipeline behaviour, only the content of an alert. This matters more than it sounds: log-injection into a SIEM query layer is a known, exploitable class of bug.

The third constraint is time. Detection value decays. A credential-stuffing alert that surfaces six hours after the fact is a compliance artefact, not a defence. So the pipeline has two competing latency budgets — a low-latency path for high-confidence, narrowly-scoped detections (impossible travel, known-bad IOC hits, privilege escalation to a break-glass role), and a batch/near-real-time path for correlation-heavy detections (beaconing patterns, slow lateral movement, statistical baselining) that need a wider observation window to avoid false positives.

The architecture starts with a log/event bus — typically Kafka or a managed equivalent — sitting between collectors and everything downstream. This decoupling matters for two reasons: it lets detection logic be replayed against historical data when a new IOC or technique is published (retro-hunting), and it lets you add or change consumers — a new detection engine, an ML scoring service, a long-term archive writer — without touching collection agents. Collectors (EDR agents, cloud log forwarders, syslog relays) write raw events in their native format to source-specific topics; a normalisation layer then maps them into a canonical schema.

The canonical schema is the single most consequential design decision in the whole pipeline, and it's worth treating as a first-class contract rather than an implementation detail. OCSF (Open Cybersecurity Schema Framework) or an internally maintained equivalent gives every event a common set of fields — actor, target, action, outcome, source telemetry type, timestamp with ingestion-lag delta — so that a correlation rule written against 'process creation' events works the same whether the underlying source is CrowdStrike, Sysmon, or a cloud workload runtime. Without this normalisation layer, every detection rule has to know about every source format, which means detection coverage degrades every time you onboard a new log source, and it degrades silently.

Schema validation belongs at the write boundary, not downstream: malformed or partially-parsed events should be quarantined to a dead-letter topic with metrics on quarantine rate per source, because a source silently failing to parse is functionally the same as a detection blind spot, and blind spots caused by parser drift are one of the most common real-world SOC failure modes — far more common than a genuinely novel attack technique.

Detection-as-code is the operating discipline that makes the rest of this maintainable: detection rules (Sigma, or a rules engine's native DSL) live in version control, go through review, and are unit-tested against a corpus of known-benign and known-malicious event sequences before deployment. This isn't process for its own sake — SOC detection content decays as attacker tooling, cloud provider log formats, and the organisation's own infrastructure change, and without tests-in-CI a 'silent break' (a rule that stops matching because a vendor renamed a field) is indistinguishable from a quiet week.

Most genuinely useful detections in this space are not exotic ML models; they're well-scoped correlation rules operating over a short, indexed time window — multiple failed auths followed by a success from a new ASN, a service account authenticating interactively, process trees that match a known living-off-the-land pattern, outbound connections to a domain that appeared in a threat-intel feed within the last 24 hours. These need a stream-processing layer (Flink, or a SIEM's native correlation engine) that can maintain sliding-window state per entity — per user, per host, per service account — cheaply, because state-per-entity at scale is where naive correlation engines fall over.

Where statistical and ML techniques earn their place is baselining and anomaly scoring on top of that correlated stream, not as a replacement for it: UEBA-style models that learn a per-entity baseline (typical login hours, typical data-egress volume, typical process ancestry) and flag deviations are good at surfacing the 'nothing matched a rule but this is still wrong' case. The trade-off is honesty about false-positive rates — an anomaly score is a prioritisation signal for triage, not an alert on its own, and treating it as one is how SOCs end up with alert fatigue and analysts who learn to distrust the tool. Threat-intel enrichment (STIX/TAXII feeds, internal IOC stores) should be joined onto events at the correlation stage, not baked into raw ingestion, so that intel updates retroactively improve scoring on already-buffered data without a schema change.

A rule match is not an alert; it's a candidate. The gap between the two is where most operationally mature pipelines differ from naive ones. Each candidate gets a severity and confidence score derived from the specific rule, the entity's baseline risk (a service account with prior incidents scores differently than a fresh laptop), and any active threat-intel correlation. Candidates are then deduplicated and grouped into cases keyed on shared entities within a time window, so that twelve related detections against the same compromised host produce one case with twelve linked events, not twelve separate tickets competing for attention — case-grouping logic is arguably as important to analyst throughput as detection accuracy itself.

Only cases that clear a severity threshold are pushed into the alerting layer proper, which fans out to a SOAR platform or ticketing system for human triage, with automated enrichment (WHOIS, asset ownership lookup, prior-incident history for the entity) attached before a human ever opens it, so the first thing an analyst sees is a decision-ready case rather than a raw log line they have to go enrich themselves. Auto-remediation — disabling a token, isolating a host — should be reserved for a narrow set of high-confidence, low-blast-radius detections with an explicit rollback path, and always logged to a tamper-evident audit trail separately from the alert itself, since 'the automation took an action' is exactly the kind of event that needs to survive an attacker who compromises the SOC tooling.

Every alert, resolved or not, closes the loop by writing its final disposition (true positive, false positive, benign-but-noteworthy) back into a feedback store keyed to the rule that fired it. That feedback is the actual measurement of whether a rule is pulling its weight, and it's what a detection-engineering team uses to tune thresholds or retire rules — without it, rule quality is a matter of opinion.

At scale the bottleneck is rarely raw ingestion throughput — a Kafka-fronted pipeline handles that reasonably well with standard partitioning by source and entity — it's stateful correlation and long-window retention. Hot storage (indexed, queryable within seconds, days-to-weeks retention) needs to be kept separate from cold storage (object storage, months-to-years retention for compliance and retro-hunting), with a tiering job that ages data down automatically; querying cold storage should be a deliberate, slower operation, not something the real-time detection path depends on. Multi-tenancy, if the SOC serves multiple business units or a security consultancy serves multiple clients, has to be enforced at the storage and query layer with per-tenant key separation, not just at the UI layer — a query-layer isolation bug here is a client-data-leak incident, not a bug ticket.

Operability failure modes deserve as much design attention as detection failure modes: a collector going silent, a normalisation job falling behind, a correlation engine's state store growing unbounded under a burst of noisy source data. Each of these is itself a detection target — pipeline health should be monitored with the same seriousness as the environment it's watching, including alerting on ingestion-lag-per-source and on rule-match-rate anomalies (a rule that goes from firing daily to firing zero times is as suspicious as one that suddenly fires constantly).

What we would deliberately avoid: a single monolithic SIEM query language as the only detection surface, because it locks detection logic to one vendor's expressiveness and makes testing hard; fully automated response on anything above narrow, pre-approved actions, because the false-positive cost of an automated action taken against production infrastructure is asymmetric and severe; and treating ML anomaly scores as alerts rather than as triage priority input, because that's the fastest route to an ignored tool. None of these choices are exotic — they're the boring, well-understood trade-offs that separate a detection pipeline analysts trust from one they route around.