OneNym.idDiscuss deployment
English

THE HARDENED STACK / Chapter 10

Secrets and Key Management

Every application and pipeline authenticates to something with a credential, and how those credentials are stored, rotated and retired decides whether an incident stays contained. This chapter explains how to move an estate away from passwords pasted into files.

This page summarises the chapter. The full text is in the book.

Why secrets in code and config are dangerous

Dev Okonkwo finds a database password committed to the trade portal's history eleven months earlier, then scans every repository and finds nine more secrets. None had been abused, but the book stresses that this was luck. A secret is any value that grants access simply by being known: passwords, API keys, tokens, connection strings, private keys and certificates.

Three problems compound. Spread: code is copied to every clone, fork, CI cache and backup, and deleting the file later does not remove it from history. Longevity: credentials in code are painful to change, so they stay unchanged for years. Visibility: anyone with repository access can read them, with no record of who did. The same applies to CI/CD variables, scheduled tasks, SaaS admin panels and shared spreadsheets of service-account passwords. The author also notes that secrets end up in code because it is the quickest way to make things work, so the goal is to make the safe alternative just as easy.

Secret managers and runtime retrieval

A secret manager such as Azure Key Vault, AWS Secrets Manager or Parameter Store, Google Secret Manager or HashiCorp Vault stores secrets encrypted, releases them only to authorised identities, logs every access and usually versions them. The application holds an identity, not the secret, and asks the vault for the value when it needs it. The value should be fetched at runtime and kept in process memory, not read during the build and baked into an image or a persistent environment variable. A secrets-store driver that injects values at start-up using the workload's identity is acceptable; the test is whether the secret survives a restart as a stored artefact.

An expert section covers caching secrets in memory for a short period to respect vault throttling, planning for the vault being unreachable, running the vault with the same availability as a database, and limiting blast radius with per-secret access, separate vaults for separate trust boundaries, and alerts on unusually broad reads.

Rotation without a maintenance window

A secret that never changes keeps any past exposure alive. Rotation should be routine and automated where the platform supports it, and immediate when someone with access leaves, a laptop is lost, a scanner finds the value or vault logs look suspicious. The key design is the two-credential model: create a new credential, store it as the current version, wait for consumers to refresh, verify it, deactivate the old one, and delete it only after a grace period. If rotation is risky, the book treats that as an architecture problem to fix. At Meridian, a quarterly password change that used to need a Saturday window became an uneventful weekday task once every consumer read from Key Vault through managed identity.

Short-lived credentials, workload identity and OIDC

The deepest fix is to remove long-lived secrets entirely. Managed or workload identity gives an application on a VM, task or pod short-lived tokens issued by the platform, so nothing is stored. OIDC federation applies the same idea to CI/CD: the pipeline presents a signed token describing itself, and the cloud exchanges it for a short session only if the issuer and subject match what it was told to trust. The chapter's AWS example restricts the role to one repository's main branch; the subject condition is what stops any other repository from assuming it. Short-lived tokens can still be stolen within their lifetime, and the assumed role should still follow least privilege.

Encryption keys, certificates and leaked secrets

Envelope encryption encrypts each piece of data with its own data key and wraps that key with a key-encryption key held in a KMS or HSM. The KEK never leaves the service, rotating it only re-wraps keys, revoking it makes the data unreadable, and every unwrap is logged. Platform-managed keys protect against disk theft; customer-managed keys let you audit and revoke but do not hide plaintext from a running service; only client-side encryption does. Losing a customer-managed key loses the data, so enable soft-delete and purge protection.

Certificates expire loudly. Keep an inventory, automate renewal with ACME or managed certificates, and monitor expiry independently of the renewal job, for example warning at 21 days; internal CAs and code-signing certificates need the same care. Finally, scan every repository's full history with tools such as gitleaks or trufflehog. When a secret is found, rotate it first, replace it with a vault lookup, purge history if feasible, then add pre-commit hooks and CI checks.

KEY POINTS

Key points

  • Scan the full history of every repository for secrets and rotate every hit before anything else.
  • Add a pre-commit secret-scanning hook and the same scan as a required CI check.
  • Move one hard-coded production secret into a secret manager this week, read at runtime via workload identity.
  • Replace one long-lived cloud key used by CI with OIDC federation scoped to your repository and branch.
  • Enable automatic two-credential rotation on at least one high-value secret.
  • Inventory TLS certificates and set an independent 21-day expiry alert for every public endpoint.

QUESTIONS

Common questions

Where should application secrets be stored instead of config files?
In a secret manager such as Azure Key Vault, AWS Secrets Manager or HashiCorp Vault, which stores them encrypted, releases them only to authorised identities and logs every access. The application should authenticate with its own workload identity and fetch the secret at runtime, keeping it in memory. Reading it at build time and baking it into an image or persistent environment variable recreates the original problem.
How can I rotate a database password without downtime?
Use the two-credential model described in The Hardened Stack. Create a new credential, store it in the vault as the current version, wait for consumers to refresh their cache, and verify it works end to end. Then deactivate the old credential, confirm nothing broke, and delete it only after a short grace period. This requires a target system that allows two active credentials at once.
How does OIDC federation remove stored cloud keys from CI/CD pipelines?
With OIDC federation the pipeline requests a signed token that describes it, for example a particular repository and branch, and the cloud exchanges that token for a short-lived role session. No long-lived cloud key is stored in the CI system. The trust policy must pin the token's subject tightly to the repository and branch you deploy from, otherwise other repositories could assume the role.
What is envelope encryption?
Envelope encryption encrypts each piece of data with its own data key, then encrypts that data key with a central key-encryption key held in a key management service such as AWS KMS or Azure Key Vault Managed HSM. The central key never leaves the service, can be rotated without re-encrypting the data, and every unwrap is logged. Revoking access to it makes the protected data unreadable.
What should I do first when a secret is found in a Git repository?
Rotate it first and assume it is compromised, because it has already been cloned and backed up. Then replace its use with a lookup from a secret manager, purge it from history if feasible, and add a pre-commit hook and a CI check so the next leak is blocked. Rewriting history first is the wrong order, since it does not undo the exposure.