Kustomize

TL;DR

• Kustomize manages environment-specific config variance without a templating language — a base of plain, valid YAML plus per-environment overlays that apply structural patches.
• Every file remains directly readable and independently kubectl apply-able; nothing is a half-rendered template.
• kubectl kustomize (or kubectl apply -k) renders bases + overlays + patches into final manifests; native support has shipped in kubectl since 1.14.
• configMapGenerator/secretGenerator auto-suffix a content hash to force rollouts on config change — a mechanism unique to this approach.
• Biggest gotcha: overlays are not self-contained. They are patches applied on top of whatever the base currently looks like, so a base refactor can silently break every overlay’s patch target.

Core Concepts

Base and overlay structure

A base/ directory holds the canonical resources and a kustomization.yaml manifest listing them. Each environment gets an overlays/<env>/ directory referencing the base and layering its own transformations (name prefixes, labels, patches, image overrides) on top.

myapp/
├── base/
│   ├── kustomization.yaml
│   ├── deployment.yaml
│   └── service.yaml
└── overlays/
    ├── dev/
    │   ├── kustomization.yaml
    │   └── patch-replicas.yaml
    └── production/
        ├── kustomization.yaml
        └── patch-resources.yaml

# base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
- service.yaml

# overlays/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
namePrefix: prod-
commonLabels:
  environment: production
patches:
- path: patch-resources.yaml
  target:
    kind: Deployment
    name: payments-api
images:
- name: myorg/payments-api
  newTag: 1.4.2

Kustomize never modifies the base directory. It resolves references, applies transformers and patches, and emits a fully-realized manifest — the base stays valid, plain, standalone YAML at all times.

Patch types

Two patch mechanisms exist, chosen based on how surgical the edit needs to be.

Strategic merge patch — the default. Merges based on Kubernetes-aware rules (matches list items by key fields like container name), behaving like a targeted, semantic YAML diff:

# patch-resources.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-api
spec:
  template:
    spec:
      containers:
      - name: app
        resources:
          limits:
            memory: "1Gi"

JSON 6902 patch — precise, path-based operations (add/remove/replace) with no merge ambiguity:

patches:
- target:
    kind: Deployment
    name: payments-api
  patch: |-
    - op: replace
      path: /spec/replicas
      value: 5

Strategic merge is preferred by default for readability; JSON 6902 is required when precise array manipulation is needed (removing a specific list item, reordering) or when the target object is a CRD lacking strategic-merge metadata (the patchMergeKey annotations that make strategic merge’s list-matching work).

Common transformers

• namePrefix / nameSuffix — prepend/append to every resource name (prod-payments-api).
• commonLabels / commonAnnotations — applied uniformly across every resource in scope.
• configMapGenerator / secretGenerator — generate ConfigMaps/Secrets from files or literals, with an automatic content-hash suffix.
• images — override image name/tag across all matching resources without touching base YAML.

configMapGenerator:
- name: app-config
  literals:
  - LOG_LEVEL=info

Why the content hash matters

Kubernetes does not restart pods when a referenced ConfigMap’s content changes while its name stays the same — the Deployment’s pod template still points at the same object reference, so nothing forces a rollout. The generated name (app-config-a1b2c3d4) changes whenever content changes, so the Deployment’s envFrom/volume reference itself changes — a genuine spec diff that Kubernetes detects and rolls out automatically. Same-name ConfigMap edits are the classic “why didn’t my config change take effect” trap this mechanism exists to avoid.

Kustomize vs Helm

Command / Configuration Reference

# Render the final manifest without applying it
kubectl kustomize overlays/production/

# Apply directly — kubectl has native Kustomize support since 1.14
kubectl apply -k overlays/production/

# Diff what would change before applying
kubectl diff -k overlays/production/

# Edit values from the CLI instead of by hand (updates kustomization.yaml directly)
cd overlays/production
kustomize edit set image myorg/payments-api=myorg/payments-api:1.4.3

Common Pitfalls

• Pitfall: An overlay’s patch silently stops applying after an unrelated base change. Why: Overlays patch whatever the base currently defines; renaming a container or moving a field breaks the patch’s target match with no explicit error. Fix: Run kubectl kustomize on every affected overlay after modifying a shared base to confirm patches still resolve.
• Pitfall: A ConfigMap update doesn’t trigger a pod restart. Why: The Deployment’s env/volume reference still points at the same unchanged object name. Fix: Use configMapGenerator so the content hash changes the referenced name automatically, forcing a rollout.
• Pitfall: A strategic merge patch fails silently or partially on a CRD. Why: Strategic merge relies on Kubernetes-aware merge keys that many CRDs never define. Fix: Use a JSON 6902 patch with an explicit path for CRD fields.
• Pitfall: Treating an overlay as a complete, standalone manifest set. Why: Overlays only contain diffs; the actual resources live in the base. Fix: Always review the rendered output (kubectl kustomize), not just the overlay directory contents, to understand the true final state.

Interview Questions

• Why does configMapGenerator append a content hash instead of keeping a stable name? — tests understanding of how Kubernetes detects spec changes to trigger rollouts.
• When would a JSON 6902 patch be preferred over a strategic merge patch? — tests depth of patch-mechanism knowledge beyond surface syntax.
• What’s the core philosophical difference between Kustomize and Helm? — tests whether the candidate understands template-free vs template-based configuration management.
• A base’s container name changes and an overlay patch stops applying with no error. What’s the root cause and how would you catch it before deployment? — tests operational awareness of overlay/base coupling and CI validation habits.