A Zero-Downtime Secrets Management and Rotation Architecture

Secrets management is usually introduced backwards: a team starts with an .env file, migrates to a shared vault when a credential leaks in a git history, and only later confronts rotation, which is the actually hard part. Storing a secret securely is a solved problem — encrypt at rest, control access, log reads. Rotating it without breaking every consumer that cached the old value is where most designs fail, because rotation is a distributed-systems problem disguised as a security feature.

The constraints we design against are consistent across clients: credentials must never live in source control, CI logs, or container images; every access to a secret must be attributable to a specific workload identity, not a shared API key; a compromised credential should have a blast radius measured in minutes, not the lifetime of the secret; and rotation must be routine enough that it happens automatically on a schedule, not as a manual fire-drill after an incident. That last point is the one organisations underestimate — a rotation process that requires a human to coordinate downtime will simply not run, and an un-rotated secret is a long-lived attack surface regardless of how well it was stored on day one.

We also treat human and machine access as fundamentally different problems. Human operators need break-glass access, MFA, and approval workflows. Services need identity-bound, short-lived credentials issued automatically at boot, with no human in the loop at steady state. Conflating the two — for example, giving an engineer's IAM user the same static database password a service uses — is the single most common design mistake we see.

At the centre sits a secrets engine (we design around HashiCorp Vault or a cloud-native equivalent like AWS Secrets Manager backed by KMS) that never stores plaintext secrets on its own disk. Every secret is protected with envelope encryption: a data encryption key (DEK) encrypts the secret value, and the DEK itself is encrypted by a key-encryption key (KEK) that lives in a hardware security module or cloud KMS and never leaves it. The vault's storage backend only ever persists ciphertext plus a wrapped DEK; compromising the storage layer alone yields nothing usable.

The KEK is the actual root of trust, and it is the one key in the system we do not rotate casually — instead we rotate it on a slower cadence (e.g. annually, or on suspected compromise) using key versioning support in the KMS, so old ciphertext remains decryptable under its original key version while new writes use the current one. This two-tier structure — a rarely-rotated root key protecting frequently-rotated data keys, which in turn protect frequently-rotated application secrets — is what makes rotation at the leaf level cheap: rotating a database password touches one DEK-wrapped value, not the whole trust chain.

Above the encryption layer sits the access-control model, and here the design choice that matters most is authenticating workloads by identity, not by a bearer token. We bind Vault (or equivalent) authentication to platform-native identity — a Kubernetes service account token via the Kubernetes auth method, an AWS IAM role via the AWS auth method, or a SPIFFE/SPIRE-issued SVID for a fully zero-trust mesh. The workload never holds a long-lived Vault token; it proves who it is using an identity the platform already vouches for, and receives a short-lived Vault token scoped to exactly the secrets its policy allows. This collapses 'who can read this secret' into the same identity system that already governs network access, rather than maintaining a parallel ACL that drifts out of sync.

The architectural decision with the biggest payoff is preferring dynamic, on-demand secrets over static, rotated ones. For a database, instead of storing a long-lived password and rotating it every 30 days, the secrets engine has a database plugin with admin credentials, and each workload requests a lease: the engine creates a brand-new database user with a randomly generated password and a TTL, hands it to the workload, and automatically revokes and drops that user when the lease expires or is explicitly revoked. Rotation stops being an event — it is simply the natural expiry of a lease, and 'rotating' a compromised credential is the same code path as normal expiry, just triggered early.

Where a target system cannot support dynamic per-request credentials — a third-party SaaS API key, a legacy service that only accepts one shared token — we fall back to scheduled rotation of a static secret, but the rotation itself follows a dual-secret window pattern to avoid downtime: the engine generates a new secret value alongside the still-valid old one, writes the new value to the target system (or the target's own rotation API, e.g. AWS IAM access key rotation, which explicitly supports two active keys), and only after confirming every consumer has picked up the new version does it deactivate the old one. Consumers are never told 'switch now'; they poll or receive a push notification and pull the current version on their own schedule, and the system tolerates both old and new secrets being valid simultaneously for a bounded overlap window.

On the read path, workloads do not fetch a secret on every use — that would make the secrets engine a single point of failure for every request in the system. Instead, a sidecar or SDK-level client fetches the secret at process start (or lease acquisition), caches it in memory only, and subscribes to invalidation: either a short TTL that forces a refetch, or an explicit revocation event the engine can push. The secret is never written to disk, environment variable dumps, or logs — it lives only in the process memory of the workload that needs it, for the duration of its lease.

The hardest failure mode is not a leaked secret — it's a rotation that goes wrong and takes down a production system, because that outcome trains operators to distrust and disable automatic rotation, which is worse for security long-term. We handle this with a validation step baked into the rotation workflow itself: after writing a new credential to the target system, the rotation job actively tests it (a real connection attempt, not just an API 'success' response) before marking the old credential for deactivation. If validation fails, the rotation aborts, the old secret remains active, and an alert fires — the system fails toward continuity of the old, known-good secret rather than toward 'security at all costs' that breaks production.

A second correctness concern is the metadata store getting out of sync with the actual state of downstream systems — for example, the vault believes a database user was rotated out, but the drop-user call failed silently and an orphaned credential is still valid. We treat this as a reconciliation problem, not a one-shot operation: a periodic sweep compares the engine's lease ledger against the actual state of each backend (list active DB users, list active IAM keys) and flags or force-revokes anything present in the backend but absent from the ledger. Idempotent revocation — safe to call twice — is a hard requirement on every backend plugin for exactly this reason.

For the audit trail, every issuance, read, and revocation is written to an append-only log that is itself tamper-evident, typically via hash-chaining each entry to the previous one (or shipping immediately to a write-once log store the application tier cannot modify). This matters less for day-to-day operations and more for incident response: after a suspected compromise, the question is always 'exactly which secrets did this identity touch, and when,' and an audit log that the compromised workload could have edited is worthless for answering it.

We chose dynamic secrets as the default and static rotation as the fallback, which means the platform depends heavily on backend plugin support — for exotic or legacy systems this means building and maintaining custom plugins, which is real ongoing engineering cost, not a one-time setup. We accept that cost because the alternative, static secrets rotated by cron jobs, degrades quietly: the rotation job breaks, nobody notices because the old secret still works, and six months later the 'rotated' secret hasn't rotated in six months.

We centralise on a single secrets engine as the trust root rather than distributing secret material across each team's own KMS keys, deliberately accepting that engine as a high-value target and a potential single point of failure. We mitigate that concentration risk with HSM-backed keys that never leave the module in plaintext, aggressive scoping of every access policy to least privilege, and a break-glass path that is itself logged and time-bounded rather than a permanent admin backdoor — but we do not pretend the concentration risk disappears, and any client of this design should weigh it against the alternative of secret sprawl, which is a worse and less visible risk in practice.

Finally, we do not build automatic rotation for every secret class from day one. Rotating the root KEK, or a secret backing a stateful system with no graceful dual-credential support (some legacy on-prem systems genuinely only accept one password and require a restart to pick up a new one), is left as a documented, tested manual runbook rather than a fragile automation that would rarely execute and would be poorly tested when it did. Automating the 80% of secrets where dynamic issuance and dual-secret rotation are natural, and being explicit about the 20% that still require a human-supervised runbook, produces a more honest and more maintainable system than automating everything and hoping the edge cases never fire in production.