Reference Architecture: Kubernetes Runtime Security & Admission Control

A Kubernetes cluster is a shared control plane with an API that will happily do whatever an authorised caller asks — schedule a privileged pod, mount the host filesystem, disable a seccomp profile, or pull an unsigned image from a public registry. None of that is a bug; it is the platform working as designed for an operator who trusts every request. In a multi-team or multi-tenant environment that trust model breaks down fast: a CI pipeline with cluster-admin-scoped credentials, a developer copy-pasting a Helm chart from a blog post, or a compromised dependency inside a build step can all produce a manifest that is syntactically valid and semantically dangerous.

The design question we are answering is not 'how do we stop a container from escaping' — that is a kernel and isolation problem — but 'how do we stop a dangerous workload from ever being scheduled, and if one gets through anyway, how do we notice while it is running.' That splits the architecture into two cooperating control points: admission control at the API server boundary, which is a policy gate over declarative state, and runtime enforcement/detection inside the node, which is a behavioural gate over what a running process actually does. Treating these as one problem is a common mistake — a validating webhook cannot see a reverse shell spawned inside a container after admission, and a runtime sensor cannot retroactively stop a bad pod spec from ever being created.

Every write to the API server passes through a chain: authentication, authorisation (RBAC), mutating admission webhooks, object schema validation, and validating admission webhooks, in that order, before anything is persisted to etcd. Native admission covers the coarse cases — Pod Security Admission, the successor to the deprecated PodSecurityPolicy, enforces baseline/restricted profiles at the namespace level (no privileged containers, no host namespaces, mandatory non-root, dropped capabilities) with a single label and no CRDs to manage. That is the floor, not the ceiling.

For anything organisation-specific — 'no image from outside our registry', 'every deployment must carry a cost-centre label', 'ingress objects must not use a wildcard host in the production namespace' — the architecture needs a policy engine running as a validating (and optionally mutating) webhook: OPA/Gatekeeper with Rego constraint templates, or Kyverno with its more Kubernetes-native YAML policy model. The choice between them is mostly organisational: Gatekeeper suits teams that already think in Rego and want policy-as-code with a general-purpose language; Kyverno suits teams that want policies reviewable by anyone who can read a Kubernetes manifest, with better native support for image verification and mutation-then-validate flows.

The architecturally important decision is failure mode. A webhook is synchronous and blocking by nature, and the API server's `failurePolicy` field decides what happens if the webhook is unreachable: `Fail` blocks all matching admission requests (safe, but an outage in the policy engine becomes a cluster-wide outage for deployments), `Ignore` lets requests through unchecked (available, but silently fail-open exactly when you need the gate most). We default to `Fail` for the smallest set of high-value policies — no privileged pods, no unsigned images, no `hostNetwork` — deployed with the webhook itself running with tight resource requests, a `PodDisruptionBudget`, and its own high-priority `PriorityClass`, so it is one of the last things evicted under node pressure. Lower-stakes advisory policies run in `Ignore`+audit mode so they never become an availability dependency.

Admission control is also the natural enforcement point for supply-chain integrity, because it is the last moment before a workload spec becomes real. A Kyverno or Gatekeeper policy can call out to Sigstore/cosign to verify that an image's signature chains back to an approved key or OIDC identity, and separately verify that an SLSA provenance attestation exists and names an expected build system before the pod is admitted. This turns 'we sign images' from a build-time nicety into an actually-enforced control — an unsigned or unattested image simply cannot be scheduled, regardless of who submitted the manifest or from where.

The subtlety is caching and latency: signature verification means a network call to a transparency log or OCI registry on every admission request, on the hot path of every pod creation including scale-up under load. We put a short-TTL verification cache in front of the webhook keyed on image digest (not tag — tags are mutable, digests are not) so a burst of replicas from the same ReplicaSet triggers one verification, not one per pod, and so registry or Rekor unavailability degrades to a bounded staleness window rather than an outage.

Admission control governs what gets scheduled; it says nothing about what a legitimately-admitted container does once code inside it starts executing — including code introduced later via a supply-chain compromise of a dependency that was clean at build time. Two complementary layers cover that gap. The first is default-deny isolation at the kernel boundary: seccomp profiles restricting the syscall surface (Kubernetes ships `RuntimeDefault` as a sane baseline; tighter workload-specific profiles are worth the maintenance cost only for high-risk workloads), AppArmor or SELinux for filesystem and capability confinement, and for genuinely untrusted or multi-tenant workloads a stronger isolation boundary than the shared-kernel container model provides at all — gVisor's user-space kernel intercepting syscalls, or Kata Containers running each pod in a lightweight VM. That last choice is a real trade-off: both add measurable startup latency and reduce compatibility with workloads that need raw syscalls (some eBPF-based sidecars, certain device access patterns), so we scope it to the tenancy boundary that actually needs it rather than applying it fleet-wide.

The second layer is runtime detection: an eBPF-based sensor (Falco is the de facto reference implementation) attached to kernel tracepoints, evaluating rules against process execution, file access, and network activity as they happen — a shell spawned inside a container that never runs one, a write to `/etc/shadow`, an outbound connection from a pod to an IP outside its expected egress set. eBPF's advantage over the older kernel-module or ptrace-based approaches is that it runs unprivileged from the workload's point of view, verified by the kernel before load, and adds negligible overhead to steady-state execution, which is what makes it viable to run on every node rather than only on ones flagged for suspicion. Detections here are necessarily after-the-fact — the process already executed the syscall by the time an eBPF probe reports it — so this layer feeds automated response (kill the pod, cordon the node, revoke associated service account tokens) rather than pretending to be a preventive control.

We would not put every policy through a `Fail`-closed webhook — the temptation to enforce everything synchronously is real, but each additional blocking webhook is another dependency that can turn a policy-engine blip into a cluster-wide inability to deploy. The discipline is to keep the `Fail`-closed set small and genuinely security-critical, and route everything else through audit-then-alert so policy iteration doesn't require touching the availability-critical path.

We also would not treat admission control as sufficient on its own, which is the most common shortcut we see: teams stand up Gatekeeper, pass a policy audit, and stop there, leaving no visibility into what admitted workloads actually do at runtime — which misses supply-chain compromises introduced after the image was built and signed, and insider threats operating within otherwise-compliant pod specs. Conversely we would not lead with heavyweight VM-based isolation (gVisor/Kata) everywhere by default; it is the right answer for genuinely adversarial multi-tenancy but is a poor default for internal platform teams' own workloads, where it mostly adds latency and operational surface for a threat model that doesn't apply. And we would not build custom Rego or Kyverno policies for controls the platform already gives you for free — Pod Security Admission's `restricted` profile covers a large fraction of what organisations reinvent badly in custom webhooks, and every policy that duplicates a built-in is one more thing to keep in sync as Kubernetes itself evolves.