Secrets Management

TL;DR

• Encryption at rest is the baseline; the real discipline is access control, audit logging, and rotation — the parts that matter when a secret leaks.
• A hardcoded credential in a Git-committed manifest is permanently recoverable from history, visible to anyone with repo read access, and has no rotation path.
• Vault’s dynamic secrets bound blast radius by issuing short-lived, per-client credentials instead of one shared static one.
• External Secrets Operator syncs external secrets (Vault, AWS Secrets Manager) into native Kubernetes Secrets so applications need zero awareness of the backend.
• SOPS encrypts individual values inside a Git-committed file, fitting GitOps workflows that need secrets version-controlled without a live external dependency at deploy time.
• Biggest gotcha: encryption alone says nothing about who accessed a secret or whether that access was legitimate — access control and audit logging are what make incident investigation possible.

Core Concepts

The baseline risk of hardcoded secrets

A database credential hardcoded as a plain environment variable in a Kubernetes manifest committed to Git is not merely “not best practice.” It is permanently recoverable from Git history even after later removal, visible to anyone with repo read access, and carries no rotation mechanism. A proper secrets manager addresses all three simultaneously: centralized access control (who can retrieve the secret), audit logging (who did retrieve it and when), and rotation (changing the credential without a manual, error-prone, multi-system update).

Dynamic secrets (HashiCorp Vault)

Vault generates short-lived, unique credentials on demand per requesting client instead of handing out one shared static credential to every consumer. This changes the actual security posture, not just the storage location:

• A leaked dynamic credential expires quickly and is scoped to a single client — blast radius is contained.
• A leaked static, shared credential grants indefinite access, and because it is shared, revoking it in response to a leak breaks every legitimate consumer simultaneously — making incident response far more disruptive.

Vault issues these dynamic credentials via secrets engines (database, AWS, PKI, etc.), each with configurable lease durations and automatic revocation on lease expiry.

AWS Secrets Manager

Provides centralized secret storage, IAM-based access control, and native rotation integrated directly into AWS. It is the default choice for an AWS-centric organization, the same way Vault is often chosen for a platform-agnostic or multi-cloud environment.

External Secrets Operator

Solves a specific Kubernetes integration gap: getting secrets from an external backend (AWS Secrets Manager, Vault, others) into applications running in a cluster, without every application independently implementing its own SDK integration and credential handling. It syncs secrets from the external source into native Kubernetes Secret objects — applications consume standard Kubernetes Secrets with zero awareness that the source of truth lives externally. This centralizes integration logic in one operator instead of duplicating it per application.

SOPS (Secrets OPerationS)

Answers a GitOps-specific requirement: encrypted secrets living directly in Git alongside Kubernetes manifests, without a live runtime dependency on an external secrets backend at deploy time. SOPS encrypts specific values within a YAML/JSON file (backed by a KMS key, PGP, or similar), so the file can safely live in a Git repository a GitOps controller reconciles from. Decryption happens at apply/reconcile time using a key the deploying system has access to — secrets stay genuinely secret in Git while fitting the “everything the cluster needs comes from Git” model.

Access control and audit logging

Encryption protects a secret from someone without the decryption key — it says nothing about who among the authorized parties actually accessed it, when, or whether that access was appropriate. Access control and audit logging are what let an organization detect misuse by someone with legitimate access, and investigate an incident after the fact with a real access trail.

Vault vs AWS Secrets Manager vs External Secrets Operator vs SOPS

Command / Configuration Reference

# Vault: enable a database secrets engine and read a dynamic credential
vault secrets enable database
vault read database/creds/readonly-role

# AWS Secrets Manager: retrieve a secret value
aws secretsmanager get-secret-value --secret-id prod/db/password

# External Secrets Operator: ExternalSecret resource syncing from AWS Secrets Manager
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
spec:
  secretStoreRef:
    name: aws-secrets-store
    kind: SecretStore
  target:
    name: db-credentials-k8s-secret
  data:
    - secretKey: password
      remoteRef:
        key: prod/db/password

# SOPS: encrypt a values file using a KMS key, then decrypt at apply time
sops --encrypt --kms arn:aws:kms:us-east-1:111122223333:key/abc secrets.yaml > secrets.enc.yaml
sops --decrypt secrets.enc.yaml

Common Pitfalls

• Pitfall: Database credentials hardcoded in a Kubernetes manifest. Why: Convenience during initial development, never revisited. Fix: Use a secrets manager from the start; treat any hardcoded credential found in Git as compromised and rotate it, since removal alone does not purge history.
• Pitfall: A shared static database credential leaks, requiring a disruptive coordinated rotation across every consumer. Why: One credential served all clients, so revocation breaks all of them simultaneously. Fix: Use dynamic, per-client credentials so revocation is scoped and low-disruption.
• Pitfall: Every application implements its own secrets-backend SDK integration. Why: No shared sync layer exists. Fix: Use External Secrets Operator (or equivalent) to centralize the integration in one operator.
• Pitfall: GitOps-managed clusters cannot store secrets safely in Git. Why: No encryption-in-Git approach was adopted. Fix: Use SOPS or a similar tool to keep secrets version-controlled and reconcilable without a runtime external dependency.
• Pitfall: A secrets store has strong encryption but no access logging. Why: Access control was treated as sufficient without audit logging. Fix: Enable audit logging on every secrets store so incident investigations can determine who accessed what and when.

Interview Questions

• A database credential was found hardcoded in a Git-committed Kubernetes manifest. Walk through full remediation, not just removing the line. — Tests whether rotation, Git history considerations, and root-cause process fixes are all covered.
• Explain, mechanically, why Vault’s dynamic secrets provide a real security advantage over static shared credentials. — Tests whether the blast-radius and revocation-disruption argument is genuinely understood.
• Design secret management for a GitOps-managed Kubernetes cluster that needs secrets version-controlled in Git. — Tests whether SOPS or an equivalent encrypted-in-Git approach is the reasoned answer.
• Why is encryption alone insufficient for a mature secrets management practice? — Tests understanding of the access-control/audit-logging gap.