Helm

Helm is the package manager for Kubernetes. Raw Kubernetes apps are dozens of YAML manifests that vary per environment — Helm bundles them into a chart: templated manifests + a values.yaml of knobs.

It solves three problems:

• Packaging — one versioned artifact (myapp-1.4.2.tgz) instead of a folder of YAML
• Configuration — one template, many environments; values swap per env instead of copy-pasted manifests drifting apart
• Lifecycle — install, upgrade, rollback, history as first-class operations with tracked revisions

One-liner: 'Helm turns a pile of YAML into a versioned, configurable, rollback-able unit of deployment.'

mychart/
├── Chart.yaml          # name, version, appVersion, dependencies
├── values.yaml         # default configuration (the public API)
├── values.schema.json  # optional: validate values
├── charts/             # dependency charts (vendored)
├── templates/
│   ├── deployment.yaml
│   ├── service.yaml
│   ├── _helpers.tpl    # named template definitions
│   ├── NOTES.txt       # printed after install
│   └── tests/          # helm test pods
└── crds/               # CRDs (installed, never upgraded!)

Key facts interviewers check:

• version = chart version (bump on any chart change); appVersion = the software version it deploys — independent things
• _helpers.tpl files starting with _ render nothing themselves; they hold named templates shared across manifests
• crds/ is special: installed on first install, silently skipped on upgrade — a real operational gotcha
• values.yaml is your chart's public API — design it like one

A release = one installation of a chart into a namespace with a name (helm install payments ./chart). The same chart can be released many times under different names.

Lifecycle:

• install — render templates with values → validate against the API server → create resources → store release record as revision 1
• upgrade — re-render with new chart/values → three-way diff (old manifest vs new manifest vs live state) → patch only what changed → revision N+1
• rollback — re-apply the manifests stored in a previous revision → creates a new revision (rollback to rev 2 from rev 5 creates rev 6 — history is append-only)

Where state lives: Helm 3 stores each revision as a Secret in the release namespace (sh.helm.release.v1.payments.v3) — gzipped, base64 release data. helm history payments reads them.

Gotcha worth naming: rollback restores manifests, not data — a rollback after a schema migration doesn't un-migrate your database.

Lowest to highest precedence:

1. Chart's values.yaml defaults
2. Parent chart overrides of subchart values (and parent values flow down scoped by subchart name)
3. -f custom.yaml files, left to right — later files win
4. --set key=value flags — highest, win over everything

helm upgrade api ./chart -f values-base.yaml -f values-prod.yaml --set image.tag=1.4.2

Here prod overrides base, and the image tag beats both.

Practical guidance:

• Maps are merged deep, but arrays are replaced whole — overriding one element of a list means restating the entire list; classic surprise
• --set syntax gets ugly fast (escaping dots, commas) — use it for one-off values like image tags in CI; keep real config in files
• helm get values <release> shows what was actually supplied (-a for computed + defaults) — the first debugging command when 'it deployed with the wrong config'

Templates are Go text/template + the Sprig function library. The built-in objects:

• .Values — everything from values.yaml + overrides
• .Release — .Name, .Namespace, .Revision, .IsUpgrade, .IsInstall
• .Chart — Chart.yaml contents (.Chart.Name, .Chart.Version, .Chart.AppVersion)
• .Capabilities — cluster facts: .KubeVersion, .APIVersions.Has "batch/v1" — for charts that adapt to cluster versions
• .Files — access non-template files in the chart (.Files.Get, .AsConfig)
• .Template — current template's .Name and .BasePath

metadata:
  name: {{ .Release.Name }}-api
  labels:
    app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
replicas: {{ .Values.replicaCount | default 2 }}

Functions you'll use daily: default, quote, toYaml, nindent, required "msg" .Values.x (fail fast on missing values), include for named templates.

Whitespace control: {{- and -}} trim adjacent whitespace — the source of 90% of 'my YAML is invalid' template bugs.

• helm install name chart — creates a new release; fails if the name already exists
• helm upgrade name chart — upgrades an existing release; fails if it doesn't exist
• helm upgrade --install name chart (a.k.a. -i) — upgrade if exists, install if not — idempotent, which is why it's the standard form in CI/CD pipelines: the same command works on first deploy and every deploy after

Useful companion flags in pipelines:

• --atomic — if the upgrade fails, automatically roll back (and --wait is implied) — no half-applied releases left behind
• --wait --timeout 5m — block until resources are ready (Deployments available, Services have endpoints); without it Helm returns as soon as objects are submitted, not healthy
• --create-namespace on first installs

Interview nuance: --wait + resource that never becomes ready + no timeout tuning = pipelines hanging for the default 5 minutes and then a pending-upgrade state to clean up — know that failure mode.

Helm 3 is clientside-only (Tiller died with Helm 2 — no in-cluster server component, no shared god-mode identity; your kubeconfig RBAC is what Helm can do).

State = Secrets in the release's namespace, one per revision:

sh.helm.release.v1.<name>.v1
sh.helm.release.v1.<name>.v2   # type: helm.sh/release.v1

Each contains the chart, the supplied values, and the rendered manifests — gzipped + base64. helm list queries these Secrets (label-filtered) in the namespace; helm get manifest/values/notes decodes them.

Operational consequences:

• Release history is namespaced — helm list -A to see everything; deleting a namespace deletes its release history
• --history-max (default 10) caps stored revisions — unbounded history on frequently-deployed services bloats etcd
• Anyone with read access to those Secrets can see your supplied values — another reason secrets don't belong in values
• Corrupted/stuck state (e.g. pending-upgrade) is fixed by deleting the offending revision Secret — the escape hatch to know before you need it

Declared in Chart.yaml:

dependencies:
  - name: postgresql
    version: "13.x.x"
    repository: https://charts.bitnami.com/bitnami
    condition: postgresql.enabled

helm dependency update resolves versions, downloads .tgz files into charts/, and writes Chart.lock (commit it — it's your reproducibility guarantee, like a package lockfile).

Configuring subcharts: values scoped under the dependency's name flow down:

postgresql:
  auth:
    database: payments
global:            # visible to ALL charts, parent and subcharts
  imageRegistry: registry.corp.io

• condition — a boolean value path toggling the dependency (postgresql.enabled: false for envs using RDS)
• alias — install the same chart twice under different names (two Redis instances)
• global — the only values visible everywhere; use sparingly, it's shared mutable state

Design caution to voice: deep dependency trees (umbrella charts wrapping umbrella charts) make values overriding and upgrades painful — beyond 2 levels, orchestrate releases with a tool (Helmfile, ArgoCD) instead of chart nesting.

The toolbox, in the order you reach for it:

1. helm template ./chart -f values.yaml — render locally, no cluster needed. Pipe to less or a file; add -s templates/deployment.yaml for one file. This is also what CI should lint
2. helm install --dry-run --debug — renders and validates against the live API server (catches unknown fields, deprecated apiVersions that pure template misses because it has no cluster to ask)
3. helm lint ./chart — static checks: chart structure, required fields, common mistakes
4. --debug on any command — prints computed values + rendered manifests on failure
5. After deploy: helm get manifest <release> (what Helm thinks it applied) diffed against live state — finds kubectl-edit drift
6. helm-diff plugin — helm diff upgrade shows exactly what an upgrade would change; belongs in every CI pipeline as a PR comment

Rendering errors decoded: nil pointer evaluating .Values.x.y = missing values path — guard with default, required, or {{ with }}; error converting YAML = whitespace/indent bug — check your nindents.

Habit that scores points: unit tests on rendered output (helm-unittest) so template regressions fail PRs, not deploys.

Classic HTTP repos: a static web server hosting chart .tgz files + an index.yaml catalog. helm repo add bitnami https://charts.bitnami.com/bitnami, helm repo update (refreshes the index), helm search repo. Hostable on S3+CloudFront, GitHub Pages, ChartMuseum, Harbor.

OCI registries (the modern path, GA since Helm 3.8): charts stored as OCI artifacts in the same registry as your images:

helm push mychart-1.4.2.tgz oci://registry.corp.io/charts
helm install api oci://registry.corp.io/charts/mychart --version 1.4.2

No repo add/index dance, same auth + replication + retention as images, and chart signing/verification rides the registry's artifact signing (cosign) — one supply chain for images and charts.

Version discipline that matters either way:

• Published chart versions are immutable — never overwrite 1.4.2; broken? ship 1.4.3
• CI publishes charts on merge with semver bumps; consumers pin versions (13.2.1, not 13.x) in prod

Direction to state: new platforms should default to OCI — HTTP repos + index.yaml are legacy with known scaling annoyances (giant indexes, stale caches).

The mechanics first: define creates a named template; template calls it but is a statement — output goes straight to the document, can't be piped, can't be indented. include is a function returning a string:

labels: {{ include "app.labels" . | nindent 4 }}

That pipe-to-nindent is why include is the only form you should use — template in YAML manifests is a latent indentation bug.

Scope passing: templates receive one argument — pass . for full context, or a dict for explicit inputs: {{ include "app.pod" (dict "ctx" . "component" "worker") }}. Explicit dicts make helpers testable and self-documenting.

A platform-grade _helpers.tpl:

• app.name, app.fullname (honoring fullnameOverride, truncated to 63 chars — DNS label limit, trunc 63 | trimSuffix "-")
• app.labels — the full app.kubernetes.io/* standard set; app.selectorLabels — the stable subset only (selectors are immutable; putting version labels in selectors breaks every upgrade)
• app.image — registry/repo/tag@digest assembly in one place
• Namespacing: prefix all names with the chart name (app.*) — named templates are global across a chart + all subcharts; two charts defining labels silently collide, last-loaded wins

Senior point: helpers are your chart's internal API — version them mentally like code, because every template and often subcharts depend on their output shape.

Mechanics: any manifest annotated helm.sh/hook: pre-upgrade (or pre-install, post-install, pre-delete, etc.) is pulled out of the normal apply and run at that lifecycle point. hook-weight orders multiple hooks; hook-delete-policy (before-hook-creation, hook-succeeded, hook-failed) controls cleanup — the default (before-hook-creation) leaves failed hook pods around for debugging but means stale objects between runs.

The classic use: a pre-upgrade Job running database migrations, weight-ordered before the app rolls.

Failure modes to name:

1. Hook Job fails → the whole upgrade fails → release may land in pending-upgrade; with --atomic you get rollback, but the migration may have half-applied — Helm can't roll back your database
2. Hooks are not part of the release diff — helm get manifest doesn't show them, rollback doesn't re-run or undo them, and they're invisible to drift tooling
3. GitOps breaks them subtly: ArgoCD translates hooks to sync-wave phases, but semantics differ (retries, deletion policies) — test the translation, don't assume
4. Hook + --wait timeout interplay: a slow migration exceeding the timeout fails the release even though the migration eventually succeeds — now state and belief disagree

When migrations shouldn't be hooks: long-running (>timeout), non-idempotent, or needing coordination (locks across replicas / multiple services) — run them as an explicit pipeline step before deploy, or an init-container gate with a migration lock table. Hooks are fine for short, idempotent, single-owner migrations; beyond that they're a scheduling hack.

Rule I give teams: every hook must be idempotent, complete in <2 min, and safe to run twice — or it doesn't ship as a hook.

Situation: CI ran helm upgrade --wait, the new pods never went Ready (bad image), the pipeline timed out and was killed — release now shows pending-upgrade, and every subsequent helm upgrade fails with 'another operation is in progress'.

Why: Helm writes the new revision Secret with status pending-upgrade before applying, and only flips it to deployed/failed on completion. Kill the process mid-flight and the lock-like status stays forever — nothing in-cluster cleans it.

Recovery options, safest first:

1. helm rollback <release> <last-good-revision> — often works directly and resets status to deployed; check helm history for the last revision marked deployed
2. If rollback also refuses: delete the pending revision Secret — kubectl delete secret sh.helm.release.v1.<name>.v<N> (the pending one). Helm now sees the previous revision as current; re-run the upgrade
3. helm upgrade --force variants and --no-hooks are situational band-aids — understand what state the cluster is actually in first (helm get manifest vs live)

Verify after recovery: live resources may be a mix of old and new (the apply half-happened) — diff live state against the last-good manifest and reconcile; --atomic on future runs prevents the mixed state.

Prevent:

• --atomic --timeout 10m in CI (auto-rollback on failure) and make CI kill-signals graceful (don't SIGKILL helm mid-apply)
• Alert on releases in pending-* status > 15 min — it's always a stuck pipeline
• This whole class of pain is a strong argument for ArgoCD-style reconciliation where desired state is continuously converged instead of imperative apply-with-lock

What they're testing: do you know release state lives in Secrets, and can you reason about cluster truth vs Helm's record rather than just retrying harder.

Helm 3 does a three-way strategic merge between: (1) the old rendered manifest (stored in the release Secret), (2) the new rendered manifest, and (3) live cluster state.

Why three-way matters: Helm 2 diffed old-vs-new only — a field someone changed live (kubectl edit) that didn't change between chart versions was silently left drifted forever. Helm 3 consults live state, so it can put back fields the chart owns even when the chart didn't change them.

The subtleties that bite:

1. Fields the chart never set are left alone — that's how HPA coexists with Helm: if your chart doesn't render replicas, the HPA-managed value survives upgrades. The moment someone adds replicas: 2 to the template, every deploy snaps replica count back and fights the HPA — the #1 real-world example. Same for webhook-injected sidecars and defaulted fields
2. kubectl edits to chart-owned fields get reverted on next upgrade — feature, not bug (drift correction), but undocumented hotfixes silently vanish on the next deploy; the 3am memory-limit bump must land in values before morning
3. Type changes (map→list) and renamed resources can confuse the patch computation — big refactors deserve helm diff review
4. Helm has no continuous reconciliation — drift is corrected only when you upgrade. ArgoCD's selfHeal exists precisely to close that gap

Debugging drift: helm get manifest (Helm's belief) vs kubectl get -o yaml (truth) vs helm template (next intent) — three-way diff by hand when something's mysteriously off.

One-liner: 'Helm owns the fields it renders — nothing more. Design charts so ownership boundaries with controllers (HPA, webhooks) are deliberate, not accidental.'

The constraint: values files live in git; release Secrets in-cluster contain all supplied values in near-plaintext. So: secret values must never enter values files or the release record.

Options:

1. helm-secrets + SOPS (+ KMS): values files encrypted at rest in git (secrets.yaml enc with KMS key), decrypted transparently at deploy time. Pros: everything in git, diffable-ish. Cons: decrypted values still land in the release Secret; CI needs KMS decrypt rights; key custody + rotation is on you; ArgoCD needs a plugin. Acceptable for small teams; scales awkwardly
2. External Secrets Operator (my default): chart templates an ExternalSecret pointing at Vault/AWS Secrets Manager; ESO materializes the K8s Secret at runtime and keeps it synced/rotated. Git and Helm hold only references — release record contains no secret material; rotation happens without redeploys; audit lives in the secret store. Cons: another controller to run; secret-store outage is a (cached, soft) dependency
3. Sealed Secrets: encrypt-to-cluster-key, commit the sealed blob. Simple, no external store — but the cluster key is now a crown jewel with a backup/DR story, re-sealing on cluster rebuilds is toil, and rotation is manual. Fine for small setups, outgrown quickly

Chart design either way: template secretKeyRefs and ExternalSecret objects, never stringData from values; required-guard the reference names; document which store paths the chart expects.

Answer that lands: 'git holds pointers, a purpose-built store holds values, the operator does delivery and rotation — and I check the release Secret in a review to prove nothing leaked into it.'

The fundamental shift: ArgoCD runs helm template and applies the output itself — there is no Helm release. No release Secrets, no helm list, no helm rollback; Argo owns state, history, and sync.

What breaks / changes:

1. Hooks — translated to Argo sync phases (PreSync/PostSync); ordering mostly works, but hook-delete-policy semantics differ and hook-weight interleaving with sync waves needs testing. Complex hook choreographies should become explicit sync-wave Jobs
2. lookup returns nothing — helm template has no cluster to query; charts using lookup for 'create-if-absent' logic silently take the absent branch every render. Charts must be lookup-free or Argo-aware
3. .Capabilities — Argo supplies API versions, but subtle differences vs live install exist; version-adaptive charts deserve a render test in CI against the target
4. Rollback = git revert — cleaner audit than helm rollback, but your muscle memory and runbooks must change
5. Random values are poison — randAlphaNum in a template renders differently every sync → perpetual drift → Argo re-applies forever. All generated values must be stable (or come from ESO/controllers)
6. helm-secrets plugins need custom Argo config (CMPs); ESO-style reference patterns just work — another point for ESO

What you gain: continuous drift correction (the three-way-merge gap closes), app-of-apps composition, and one deployment model for helm/kustomize/plain YAML.

Recommendation I give: treat Helm as a templating format under GitOps — charts must render deterministically from values alone. Test that property in CI: render twice, diff must be empty.

The behavior: files in crds/ install on first helm install — and are then never touched again: not upgraded, not diffed, not deleted with the release. Helm hard-codes this conservatism because CRD changes are cluster-global and destructive mistakes delete every CR instance with the definition.

Why that's a trap: operators evolve their CRDs every release (new fields, new versions). Chart 2.0 ships CRD v1beta2 schema in crds/, your cluster keeps the 1.0 schema silently — the operator then creates CRs the stale CRD rejects, or defaults are missing. Failure is quiet and appears as operator bugs, not install errors.

Real management options:

1. Separate CRD chart/step (the standard): myoperator-crds chart (CRDs as regular templates, not crds/) applied first — upgrades work because they're normal resources; the operator chart depends on it being present. Argo: CRD app at an earlier sync wave
2. Templates + annotations: CRDs in templates/ with helm.sh/resource-policy: keep — upgradeable AND survive release deletion (protecting CR data). This is what many mature charts (cert-manager pattern) effectively do behind an installCRDs flag
3. Never let two releases own the same CRDs — cluster-scoped, single owner, period. Multi-tenant clusters: platform team owns all CRDs centrally

Upgrade discipline: CRD schema changes reviewed like API changes (they are API changes) — additive fields fine; version bumps need conversion webhooks and a storage-version migration plan; deletion needs a data-loss sign-off (deleting a CRD cascades to every CR).

Interview one-liner: 'crds/ means install-once-forget-forever. Anything with a living operator needs CRDs managed as first-class, upgradeable resources with a single named owner.'

The values file is a product API with 30 customers — design it like one.

Principles:

1. Safe defaults, minimal required input: a team should deploy with ~5 lines (image, resources maybe); everything else has production-sane defaults (probes on, PDB on, security context restricted). required only for what genuinely can't default
2. Progressive disclosure: common knobs top-level and flat (replicas, resources); expert knobs nested under explicit sections (advanced.topologySpreadOverrides). Alphabetical dumps of 400 keys are hostile
3. Escape hatches, bounded: podAnnotations, extraEnv, extraVolumes, extraContainers — freeform lists merged into rendered output. Without them teams fork the chart the first time you're missing a knob; with them, forks ~never happen. Balance: don't add rawPodSpecOverride — that's abdicating the chart's job
4. values.schema.json — non-negotiable at this scale: types, enums, required combos validated at render; 'replicas: "two"' fails in PR CI, not at 2am. Schema doubles as documentation
5. Conventions matching the ecosystem: image.repository/tag/pullPolicy, resources as native K8s shape, standard label helpers — muscle memory from public charts should transfer
6. Compatibility discipline: values keys are API — renames/removals are major version bumps with deprecation warnings first (fail/tpl notice when old key present). Changelog per release

Governance that makes it stick: chart owned by platform with a CODEOWNERS review path, golden-file render tests for representative team configs (a values change that alters someone's rendered output shows in the PR diff), and an office-hours channel — the chart is a product, treat requests like feature requests.

Metric of success: fork count zero, and P50 time-to-first-deploy for a new team measured in minutes.

Control structures:

• {{ if .Values.ingress.enabled }} ... {{ end }} — conditionals; combine with and/or/not/eq
• {{ with .Values.nodeSelector }} — scoped . rebinding + implicit nil-guard: block skipped entirely if value is empty — the idiomatic 'render this section only if configured'
• {{ range .Values.hosts }} — iterate lists (. = item) or maps (range $k, $v := ...). Inside range/with, . is rebound — reach the root via $ ($.Release.Name); forgetting $ inside a loop is the classic template bug

tpl — render strings as templates: lets values themselves contain template expressions:

# values.yaml
externalUrl: "https://{{ .Release.Name }}.corp.io"
# template
url: {{ tpl .Values.externalUrl . }}

Essential for reusable values across environments; costs render time and makes values Turing-uncomfortable — use for interpolation, not logic.

Sidecars from values:

# values.yaml
sidecars:
  - name: log-shipper
    image: fluent-bit:2.2
    resources: {requests: {cpu: 50m}}

# deployment.yaml template
containers:
  - name: app
    ...
{{- range .Values.sidecars }}
  - name: {{ .name }}
    image: {{ .image }}
    {{- with .resources }}
    resources: {{ toYaml . | nindent 6 }}
    {{- end }}
{{- end }}

Or accept fully-formed container specs and pass through: {{- toYaml .Values.extraContainers | nindent 2 }} — less validation, maximum flexibility; pick per audience.

Senior habit: toYaml | nindent for any structured passthrough (never hand-assemble YAML in loops), and unit-test the empty-list, one-item, and many-item renders.

What rollback does: re-applies the stored rendered manifests of the target revision (as a new revision), using the same three-way merge as upgrade. Kubernetes then reconciles — Deployments roll pods back to the old spec.

What it does NOT restore:

1. Data & schemas — the database migrated by v5's hook stays migrated; rollback to v4's code against v5's schema is only safe if migrations were backward-compatible (expand-contract discipline is what makes rollback real)
2. Hooks don't re-run by default on rollback — anything hooks did stays done
3. Out-of-band state — resources created by operators/controllers in response to your app, external side effects (queues, caches, feature flags)
4. PVC contents — the manifest reverts; the bytes on disk don't
5. Resources removed in the newer revision get re-created; resources added get deleted — check nothing now depends on them

The runbook:

1. helm history <release> → identify last-good (status deployed, known-good app version)
2. Pre-check: did any revision between now and target include migrations or contract changes? (This is why deploy PRs tag 'contains-migration' — the rollback decision needs it at 3am)
3. helm rollback <release> <rev> --wait --timeout 5m
4. Verify: pods on old image (kubectl get pods -o jsonpath image check), golden signals recovering, error budget burn stopping
5. If schema blocks code rollback: roll forward instead (revert the code commit, ship v6) — rollback is one tool, not the goal; restoring service is
6. Post-incident: the revision that failed stays in history for forensics (helm get values <release> --revision N)

Senior close: 'rollback restores manifests. Whether it restores service depends on engineering discipline you exercised before the incident — compatible migrations, no hidden hook side effects, stateless-by-default design.'

The test pyramid for charts:

1. Static (every PR, seconds):
   • helm lint + helm template render across a values matrix (default, minimal, prod-like, every feature flag on) — rendering must not error
   • values.schema.json validation tests — invalid inputs must fail
   • Rendered output piped through kubeconform (schema-validate against target K8s versions) and policy checks (Kyverno/OPA CLI — 'no latest tags, resources set, PDB present')
2. Unit (every PR, seconds): helm-unittest — assert on rendered YAML: 'given ingress.enabled=true and 2 hosts, exactly 2 rules render with these annotations'. Golden-file snapshots for whole-manifest regression — a values refactor that changes anyone's output shows as a diff
3. Integration (every PR or merge, minutes): chart-testing (ct lint + ct install) against an ephemeral kind cluster — real install, wait for readiness, run helm test pods (smoke: endpoint answers, migration ran), then ct also tests upgrade from the previous released version — the test everyone skips and the one that catches immutable-field breaks and CRD drift
4. Release gate: version bumped (ct enforces), changelog entry, provenance signed, then publish to OCI

The two failure classes worth naming because tests catch them:

• Render-time: nil pointers on unusual values combos — matrix rendering finds these
• Upgrade-time: selector/immutable-field changes, CRD skew — only upgrade tests find these

Cultural point: charts used by 30 teams get the same CI rigor as a service — 'it's just YAML' is how platform outages happen. Time cost: the full suite runs in <10 min and pays for itself the first prevented broken release.

Why: several fields are immutable post-creation — most famously Deployment spec.selector, StatefulSet volumeClaimTemplates, Service clusterIP, Job spec.template. If your chart derives selector labels from something that changed (chart name refactor, added version label to selectorLabels helper), the patch tries to mutate an immutable field and the API server rejects it. Helm surfaces the raw error mid-upgrade.

Recovery options, ordered by blast radius:

1. Revert the label change (zero impact): restore the old selector in the template — selectors are forever; the standard is a minimal, stable app.kubernetes.io/name + instance set that never changes. This is why selectorLabels and labels are separate helpers
2. Delete + recreate the Deployment, keep pods serving (near-zero impact, careful hands): kubectl delete deployment --cascade=orphan — pods keep running unowned; upgrade creates the new Deployment whose selector adopts (or replaces via rolling update) the orphans. Verify selector actually matches orphaned pods or you'll have zombie pods forever
3. Accept a recreation window (downtime for that workload): delete the Deployment normally, helm upgrade recreates — fine behind a queue or for pre-prod
4. Blue-green at the release level (zero downtime, more work): install as a new release name, shift traffic at the Service/ingress, remove the old release — for when the rename is deliberate and permanent

StatefulSet variant is worse: volumeClaimTemplates immutability means options involve orphan-delete with PVC retention and careful re-adoption — data on the line; rehearse in staging, always.

Prevent: helm-unittest snapshot on selectors (any PR changing them fails loudly), and upgrade tests from the last released chart in CI — this is exactly the class ct install --upgrade exists for.

One-liner: 'selectors are a contract with every running pod — chart refactors change labels, never selectorLabels.'

Umbrella chart (one parent chart, 12 subchart dependencies):

• One helm install deploys everything; values in one tree; version-locked composition (the umbrella version pins all parts)
• The pain at scale: one release = all-or-nothing upgrades (one service's bad rollout fails/blocks the platform deploy), values overriding through 2+ levels is miserable, no per-service history/rollback, and render time balloons
• Right for: tightly-coupled things versioned as a unit (an operator + its CRDs + dashboards), or vendor distribution of a suite

App-of-apps (ArgoCD): a root Application generating 12 child Applications (via ApplicationSet or a directory of app manifests):

• Per-service releases: independent sync, health, rollback, and history; sync waves order dependencies (infra → data → services); one bad service degrades one Application, not the platform deploy
• Composition lives in git structure, not chart nesting — much easier to reason about ownership (each child app can point at a team's own chart + values)
• Right for: anything platform-shaped under GitOps — this is my default

Helmfile: declarative multi-release orchestration without Argo — helmfile.yaml lists releases, values layering per env, helmfile diff/apply in CI:

• Keeps native Helm releases (real helm rollback), strong env layering, no controllers needed
• Right for: CI-driven shops not (yet) on ArgoCD, or bootstrap phases before Argo exists

My composition for 12 services: ArgoCD app-of-apps; each service = one Application → team-owned chart (built on the shared library chart) + env values; sync waves: CRDs/operators (wave 0) → databases/queues (wave 1) → services (wave 2) → ingress/routing (wave 3). Umbrella charts only where components are genuinely inseparable.

One-liner: 'umbrella charts couple lifecycles; app-of-apps composes them. Compose at the release layer, version at the chart layer.'

What it does: queries the live cluster at render time — lookup "v1" "Secret" "ns" "name" returns the object (or empty dict). Enables 'create-once' patterns: generate a random password only if the Secret doesn't already exist, adopt existing resources, adapt to what's installed.

Why it's dangerous:

1. Renders become non-deterministic — output depends on cluster state at that moment; the same chart+values renders differently across runs. Diffs in CI lie; reproducing a deploy becomes archaeology
2. GitOps silently breaks it — ArgoCD renders with helm template (no cluster) → lookup returns empty → your 'reuse the existing password' branch instead regenerates on every sync → perpetual drift, rotated credentials, broken sessions. This exact bug ships constantly via public charts
3. helm template and --dry-run (client) also return empty — the code path you test isn't the one that runs
4. RBAC coupling: render now requires read permissions on arbitrary objects

Deterministic alternatives:

• Random credentials: don't generate in templates at all — ESO/Vault generates and delivers; or a one-time seed Job with helm.sh/resource-policy: keep; or (bounded) randAlphaNum + resource-policy: keep + immutable Secret so the value is generated once and never re-rendered into
• 'Does X exist' logic: make it an explicit values flag (existingSecret: name) — the operator tells the chart, the chart doesn't guess. This is the pattern mature charts (Bitnami's existingSecret) converged on
• Adoption/conditional installs: move to an operator or a pre-install Job where querying the cluster is honest runtime behavior, not render-time magic

Policy I set: lookup is banned in charts that ship to GitOps-managed clusters; CI greps for it. Charts must be pure functions of values.

One-liner: 'lookup turns a template into a program with hidden inputs — great demo, terrible contract.'

Situation: 30 teams, each with hand-rolled charts — no PDBs, inconsistent labels (cost attribution impossible), three teams with prod-breaking probe misconfigurations in one quarter, security context roulette.

Task: one golden path to production without freezing teams or triggering a platform-vs-teams war.

Action:

1. Library chart (type: library) exposing named templates — corp.deployment, corp.service, corp.hpa, corp.pdb — encoding the paved road: standard labels (cost tags mandatory), restricted security context, probes required, topology spread, PDB defaults. A team's chart became ~30 lines: include the templates, supply values
2. Escape hatches by design: extraContainers, extraEnv, podAnnotations, and per-template override points — the rule was 'the paved road covers 90%; the other 10% extends, never forks'
3. Adoption as a product launch, not a mandate: migrated 3 friendly teams first (we did the PRs), published before/after diffs showing deleted YAML (~400 lines → 30), then internal docs + office hours. Made it the default in the service scaffolder so new services started on it for free
4. Version discipline: semver, deprecation warnings rendered as NOTES + CI warnings two minors before removal, golden-file render tests against every consuming team's values (their PRs run our tests; our releases run against their configs)
5. Enforcement last, carrots first: after 80% organic adoption, admission policies (Kyverno) required the standard labels/PDBs — by then that meant 'use the library or replicate its output', not 'rewrite your deploy'

Result: 27/30 teams in 2 quarters; probe/PDB incident class → zero; cost attribution coverage 40%→98%; new-service time-to-prod from ~2 weeks to 2 days.

What I'd do differently: ship values.schema.json from day one (we added it after the first wave of typo bugs), and version the values contract separately from the template internals — early refactors churned team values files more than necessary, which cost trust we had to re-earn.

What they're testing: platform-as-product instincts — adoption mechanics and compatibility discipline over templating cleverness.

The problem: a third-party chart (vendor, community) almost fits — but you need to add a sidecar, force your registry mirror, inject labels, or fix a hardcoded field the chart exposes no value for. Options historically: fork the chart (now you own drift against upstream forever) or beg upstream for a knob.

Post-renderer = the third option: any executable that receives Helm's fully-rendered manifests on stdin and returns modified manifests on stdout, running before apply:

helm upgrade -i grafana grafana/grafana --post-renderer ./kustomize-wrapper.sh

The canonical wrapper runs kustomize: rendered output becomes a kustomize base; your patches (strategic-merge or JSON6902) layer on top — 'add this initContainer to the Deployment named X', 'set imagePullSecrets everywhere', 'prepend registry mirror to every image'.

When it beats forking:

• Upstream chart updates flow freely — helm repo update + version bump; your patches re-apply to the new render (a fork requires re-merging every upstream release)
• Patches are small, explicit, reviewable — the delta is the documentation of your customization
• Org-wide invariants (labels, registries, security contexts) applied uniformly over charts you don't control

When NOT:

• The chart has a supported value for it — always prefer the front door; post-render patches on paths that upstream refactors will silently misfire (patch targets by name/kind — renames break them)
• Deep structural changes — if you're rewriting half the render, you want your own chart
• Patch fragility discipline: pin chart versions, and CI-test the patched render against every chart bump so upstream refactors fail loudly, not silently

GitOps note: ArgoCD supports Helm+kustomize combos natively (or via CMPs) — same pattern, declarative.

One-liner: 'post-renderers give you surgical control without forking custody — pay for it with patch-fragility you must test against every upstream bump.'

Two distinct ceilings — know both:

1. Render time. Costs come from: template complexity (nested includes in range loops over big lists — O(items × template depth)), tpl on large strings (re-invokes the template engine per call — tpl inside loops is the classic hot spot), giant values trees (deep merges of multi-thousand-line values), and umbrella charts rendering dozens of subcharts serially.

Diagnose crudely but effectively: time helm template with subsets — comment out template groups / dependency conditions until the slow one shows; check for tpl-in-range patterns first.

Fixes: hoist tpl/include results into variables outside loops ({{ $img := include "app.image" . }}), replace generated-per-item boilerplate with one template + item dict, split mega-charts into composed releases, and cap values-file size (values as data, not as a config database — 5,000-line values files usually mean the chart is doing an orchestrator's job).

2. Release Secret size. Helm stores chart + values + rendered manifests, gzipped, in one Secret — etcd caps values at ~1MB. Blowing it fails the upgrade outright with a cryptic 'Secret is invalid' / request-too-large error.

What bloats it: huge rendered output (hundreds of manifests), files in the chart (.Files pulling binaries/dashboards — a 2MB Grafana dashboard JSON bundle is a classic), and fat values (base64 blobs, embedded certs).

Fixes: move blobs out of the chart (dashboards from a sidecar/configmap-generator or git-sync; certs from cert-manager; data from object storage), split one release into several (each gets its own 1MB budget), aggressive .helmignore, and --history-max 5 so etcd holds fewer copies.

Structural smell to name: charts approaching these limits are usually one chart doing a platform's job — the fix is composition (multiple releases orchestrated by Argo/Helmfile), not compression tricks.

What they're testing: you know Helm's physical limits (etcd 1MB, render = template-engine cost) and read them as architecture feedback, not annoyances.

The problem: helm install on resources that already exist fails with 'resource already exists and cannot be imported into the current release' — Helm refuses to adopt what it didn't create. Common during: migrating hand-applied YAML to charts, splitting/renaming releases, or DR rebuilds where objects survived but release Secrets didn't.

The adoption mechanism (Helm 3.2+): Helm identifies ownership via metadata — set it and Helm will adopt:

kubectl label  <kind>/<name> app.kubernetes.io/managed-by=Helm
kubectl annotate <kind>/<name> meta.helm.sh/release-name=<release>
kubectl annotate <kind>/<name> meta.helm.sh/release-namespace=<ns>

Then helm upgrade --install succeeds and the resource joins the release. Script it over the full resource list for a real migration.

Migration runbook (hand-YAML → chart):

1. Build the chart; helm template and diff rendered output vs live objects until the diff is only noise (managed fields, defaults) — the chart must reproduce current state before it manages it; any real diff will be applied at adoption time (surprise rollout)
2. Label/annotate all target resources (dry-run the script; verify with a kubectl get -o yaml | grep meta.helm sweep)
3. helm upgrade --install with --dry-run first, then real; watch for unexpected patches
4. Immutable-field collisions (selectors you can't reproduce) → the orphan-delete-recreate dance per resource, scheduled deliberately

Reverse direction bonus (freeing resources from a release): helm.sh/resource-policy: keep annotation → resource survives helm uninstall — used for PVCs, CRDs, and when dissolving a release without killing its objects.

Modern note: recent Helm also has --take-ownership on upgrade to force adoption — know it exists, and prefer the explicit label/annotate flow in production migrations because it's auditable, per-resource, and reversible.

What they're testing: do you know ownership is just metadata, and do you have the discipline to make the chart render match reality before flipping management.

The threat model: a chart is arbitrary code execution against your cluster's API — it renders whatever manifests it wants (privileged DaemonSets, RBAC grants, mutating webhooks), pulls images you didn't inspect, and hooks run Jobs. Treat third-party charts like third-party dependencies: assume compromise happens (typosquatted repos, hijacked upstreams, malicious versions have all occurred).

Verification mechanisms:

1. Classic provenance: .prov files — PGP-signed chart checksums; helm verify / --verify. Real but weakly adopted; key distribution is DIY
2. OCI + cosign (the current answer): charts as OCI artifacts signed like images — keyless cosign signatures tied to the publisher's CI identity; verified at admission (Kyverno/policy-controller checks chart and image signatures). One supply chain for both artifact types

The vetting pipeline I run for third-party charts:

1. No direct-from-internet installs in prod, ever. Charts are pulled once into an internal OCI registry (pull-through with quarantine) — pinned by version + digest
2. Automated review on ingest: helm template with prod-like values → scan rendered output — cluster-role grants? hostPath/privileged? image sources (rewrite to internal mirror)? hooks doing anything odd? Policy CLI (Kyverno test) flags violations
3. Human review for high-privilege charts (operators, anything with RBAC beyond its namespace) — the RBAC it requests IS the blast radius you're accepting
4. Version bumps arrive as PRs with rendered-output diffs (not just Chart.yaml diffs) — reviewing 'chart 5.1→5.2' is meaningless; reviewing 'the render now adds a ClusterRoleBinding' is the actual change
5. Runtime: admission requires signatures + internal-registry images, so an unvetted chart physically can't deploy

Values hygiene as supply chain: your values for third-party charts are config-as-code too — a values change enabling createClusterRole: true deserves the same review weight as the chart bump.

One-liner: 'a chart install is kubectl apply of someone else's intentions — mirror it, render it, diff it, sign it, and let admission enforce all of that so the process doesn't depend on memory.'

The layering structure (per service):

values.yaml            # chart defaults: prod-safe, works everywhere
values-shared.yaml     # org/app constants across envs
values-dev.yaml        # small resources, debug on, fast probes
values-staging.yaml    # prod-shaped, scaled down
values-prod.yaml       # real resources, real replicas, alerts on

Applied as -f values-shared.yaml -f values-<env>.yaml (or Argo Application per env listing the same stack). Rules that keep it sane:

1. Env files contain only deltas — full copies per env is how staging silently stops resembling prod. Small env files are the health metric: if values-prod.yaml is 300 lines, defaults are wrong
2. Defaults are prod-safe — a missing override should fail safe (restricted security, PDBs on, small-but-real resources), never 'debug mode because dev was the default'
3. Structural parity enforced: staging differs from prod in scale, never in shape — same flags, same topology, same dependencies (smaller). Every prod-only incident traces to a shape difference; CI can diff the two files and flag non-scale keys
4. No env conditionals in templates — {{ if eq .Values.env "prod" }} buries environment logic where it can't be diffed; behavior differences must be visible as values differences

Promotion mechanics: an image/chart version promotes by PR that copies the pin from staging file → prod file — the PR diff is the promotion review. Automation (Argo Image Updater, a promote bot) writes the PRs; humans (or automated gates: staging soak, error budget green) merge them. Never promote by rebuilding — the artifact that soaked in staging is byte-identical to what ships to prod (digest-pinned).

Prod-surprise prevention: render all envs in CI on every chart/values PR (helm template matrix + kubeconform + policy) — a change intended for dev that alters prod's render shows up as a prod-file diff in review; golden snapshots make it unmissable.

One-liner: 'environments should differ by numbers, not by shape — and every promotion should be a reviewable one-line diff of a digest.'

Why the gap exists: helm template is pure text generation — it validates Go template syntax and produces YAML, but knows nothing about your cluster. Everything the API server enforces is invisible to it.

The error classes that sail through rendering:

1. Schema violations — misspelled fields (replica:), wrong types, fields in the wrong place: rejected by server-side validation (or worse, silently dropped on clusters/paths without strict validation — the deploy 'succeeds' minus your setting)
2. Deprecated/removed apiVersions — renders happily, API server has no such endpoint. The recurring K8s-upgrade breakage class
3. Admission rejections — PSA restricted denying your pod spec, Kyverno/OPA policies (missing labels, no resources), LimitRange/ResourceQuota violations
4. Webhook dependencies — cert-manager not installed → your Certificate CR has no CRD to land on; sidecar injector down → rejected or unmutated pods
5. Immutable-field conflicts and cross-resource constraints — selector changes, name collisions with resources owned by another release, references to Secrets/ConfigMaps that don't exist (pods stick in CreateContainerConfigError after a 'successful' apply)
6. .Capabilities divergence — template offline assumes API versions the real cluster doesn't have

Shifting them left, in CI:

• kubeconform against pinned schemas for every supported K8s version — kills class 1-2 in seconds, including CRD schemas if you export them
• Policy engines offline: kyverno apply / OPA eval against rendered output with the same policies prod enforces — class 3 becomes a PR failure
• helm install --dry-run=server in a pipeline stage with cluster access — real server-side validation + admission (server dry-run runs webhooks!) without persisting; the closest thing to the truth short of deploying
• Ephemeral kind/vcluster install tests (chart-testing) — catches classes 4-6: real CRDs, real ordering, real readiness
• Pin and test against the versions you actually run, including the next one before cluster upgrades

One-liner: 'helm template proves your YAML is well-formed; only the API server proves it's valid — so put a real (or dry-run) API server in the pipeline before one surprises you in prod.'

The two versions and their contracts:

• version (chart semver) — versions the deployment logic and values API: patch = template fix, no values change; minor = new optional values/features; major = breaking values change or behavioral break (renamed keys, changed defaults with blast radius, selector changes)
• appVersion — the software being deployed; informational, decoupled. A chart minor can bump appVersion; an app major usually forces at least a chart minor

Release discipline:

1. Immutability: published versions never change — repin, never republish. CI enforces version-bump-on-chart-diff (chart-testing does this)
2. Changelog per release with values-API changes explicitly listed (added/deprecated/removed keys) — machine-readable enough for a values-migration checker (artifacthub annotations serve this)
3. Deprecation pipeline: old key still works + renders a loud NOTES/warning for ≥2 minors → removed only in the next major. Templates support both keys during the window (coalesce .Values.newKey .Values.old.key)
4. Renovate/dependabot on consumer repos so bumps arrive as PRs with rendered diffs — consumers on autopilot for patches/minors, deliberate for majors

Safe passage across a breaking change (say 2.x → 3.0 renames the ingress values block):

1. Ship 2.9 with the new key supported and deprecation warnings on the old — consumers migrate values while still on 2.x, verified by warnings disappearing
2. Provide a values migration script or explicit mapping table in the 3.0 changelog; golden-render tests on both shapes
3. Consumers upgrade to 3.0 as a values-only diff already rehearsed — the chart bump itself becomes near-zero-risk; upgrade tests (ct install --upgrade from 2.9) prove no resource replacement surprises (selectors! — a breaking selector change is a 'delete and recreate' migration, document it as such with a maintenance-window flag)
4. Support window: 2.x gets security patches for N months so nobody upgrades under duress

What they're testing: do you treat the values file as a versioned public API with deprecation mechanics, and do you know a chart major is a migration you engineer, not a number you bump.

What Helmfile is: a declarative spec (helmfile.yaml) for many Helm releases — which charts, versions, namespaces, values layers — with helmfile diff / apply / sync / destroy operating on the whole set or a selection. It's release orchestration while keeping native Helm releases underneath (real helm list/rollback still work — unlike Argo's template-and-apply).

The structure that works:

environments:
  dev:     { values: [envs/dev.yaml] }
  prod:    { values: [envs/prod.yaml] }
---
releases:
  - name: ingress-nginx
    chart: ingress-nginx/ingress-nginx
    version: 4.10.0
    namespace: ingress
    values: [values/ingress/common.yaml.gotmpl, values/ingress/{{ .Environment.Name }}.yaml]
  - name: payments
    chart: oci://registry.corp.io/charts/service
    version: 2.3.1
    labels: { team: payments, tier: app }
    needs: [data/postgres]        # ordering DAG

The features that earn it a place:

• Environment layering with templated values files (.gotmpl — values rendered before Helm sees them) — env inheritance cleaner than bash-assembled -f chains
• needs — a dependency DAG across releases (infra → data → apps) without umbrella-chart coupling
• Selectors: helmfile -l team=payments apply / -l tier=infra — surgical operations on a slice of the platform
• helmfile diff (via helm-diff) as the PR gate: the rendered change-set for the whole environment in one view

Where it sits vs ArgoCD: Helmfile is imperative-when-invoked (CI runs it; no continuous reconciliation, no drift correction, no UI) but has zero cluster footprint and keeps Helm-native semantics. Argo is the end-state for continuous GitOps; Helmfile shines for: bootstrap (installing Argo itself + prerequisites), platforms not ready for controllers, ephemeral env stamping in CI, and teams that want git-reviewed diff output with native rollback retained.

Honest failure mode: nothing reconciles between CI runs — kubectl drift persists until the next apply; pair with periodic helmfile diff runs alerting on drift if you stay pre-Argo.

One-liner: 'Helmfile is terraform plan/apply ergonomics for a fleet of Helm releases — the right bridge until (and underneath) ArgoCD, not a competitor to it.'

Situation: kube-prometheus-stack major bump — the notorious kind: CRD changes, renamed values, potential immutable-field conflicts on StatefulSets — destined for 40 production clusters.

Task: ship it with zero monitoring blackouts (it IS the monitoring) and a rollback story at every step.

Action:

1. Read before rendering: upstream changelog + migration guide; identify the breaking classes — CRD schema bumps (need kubectl apply of new CRDs first, since Helm won't touch crds/), values renames, resource renames (immutable selectors → delete/recreate plans)
2. Render-diff at scale: CI job renders old vs new against every cluster's values (we keep per-cluster values in git — this is why) and produces per-cluster diffs; a script buckets clusters by diff-shape. 40 clusters usually collapse into 3-4 cohorts. Surprises live in the outlier bucket — those get human review first
3. Values migration as its own PR: map renamed keys, run both-shape golden tests, land it on the old chart version where tolerated (deprecation-compatible), so the version bump PR is version-only
4. CRD pre-step in the pipeline: apply new CRDs explicitly, verify existing CRs still validate against new schemas (a dry-run pass over live CRs catches conversion breaks before the operator does)
5. Canary cohort: 2 low-blast-radius clusters → upgrade → soak 48h watching the meta-monitoring (the second monitoring stack that watches the first — you have one, right?) for scrape gaps, rule evaluation errors, silenced-alert diffs
6. Wave rollout: cohort by cohort over a week, each wave gated on the previous one's soak; the StatefulSet-recreation clusters scheduled in maintenance windows with PVC retention verified
7. Rollback rehearsed: for this stack, rollback = chart version revert + old CRDs are NOT restorable safely once CRs use new fields — which is exactly why CRD-compatibility verification (step 4) happens before any cluster, and why the canary soaks long

Result pattern: the render-diff cohorting turns '40 upgrades' into '4 reviewed changes + mechanical repetition'; the incidents this process has caught were 100% in the outlier bucket and the CRD dry-run.

What they're testing: fleet thinking (cohorts, waves, meta-monitoring), the CRD blind spot, and whether 'renovate merged it' is your process or the start of your process.

First, challenge the premise (they want this): one chart per shape is often better than one chart with modes — modes multiply the test matrix and every conditional is a place two modes can drift. The single-chart answer is right when the shapes share 80%+ of their spec (same image, env, volumes, config) and teams switch between them (a worker that's a Deployment in streaming mode and a CronJob in batch mode). Otherwise: three thin charts on a shared library.

If one chart is right, the structure that stays maintainable:

1. Discriminator value with schema enforcement:

workload:
  kind: Deployment   # enum: Deployment | StatefulSet | CronJob (values.schema.json)

2. Share the pod, not the workload: the podTemplate is 90% of the YAML and identical across modes — extract it once:

{{- define "app.podTemplate" -}}
metadata: { labels: {{ include "app.selectorLabels" . | nindent 4 }} }
spec: { containers: [...], volumes: [...] }
{{- end }}

3. One thin template per kind, each ~20 lines wrapping the shared pod template with kind-specific fields — deployment.yaml (strategy, replicas), statefulset.yaml (serviceName, volumeClaimTemplates), cronjob.yaml (schedule, jobTemplate wrapping the same pod). Guard each with {{- if eq .Values.workload.kind "StatefulSet" }}. No mega-template with nested kind-conditionals — per-file guards keep each shape readable and diffable
4. Conditional accessories follow the discriminator: headless Service only for StatefulSet; HPA only for Deployment (schema rejects hpa.enabled with CronJob — invalid combos fail at render, not at deploy)
5. Test matrix = modes × key flags: helm-unittest snapshots per mode; ct install for each mode in CI. The moment a fourth mode or mode-specific values sections start breeding (statefulsetOnlySettings:), that's the signal to split charts

The maintainability rule: conditionals select between whole files, shared logic lives in named templates, and the schema makes illegal states unrepresentable.

One-liner: 'share the pod template, isolate the workload wrappers, let the schema police the combinations — and stay honest about when three small charts beat one clever one.'

Small surfaces, outsized impact on operability — and interviewers notice people who care about them.

NOTES.txt — rendered (it's a full template) and printed after install/upgrade; helm get notes retrieves it later. Great charts use it as the post-deploy runbook:

• Contextual access instructions — computed from actual values: if ingress enabled, print the real URL from .Values.ingress.hosts; if ClusterIP, print the exact port-forward command with release name interpolated
• State-aware warnings: {{ if not .Values.persistence.enabled }}WARNING: running with ephemeral storage — data is lost on pod restart{{ end }}; deprecation notices for old values keys (the deprecation pipeline's delivery mechanism)
• Next steps: how to get the generated admin password (the exact kubectl get secret command), what to configure next, where the dashboards are
• Anti-pattern: ASCII-art banners and static marketing text — notes should be computed, or they're noise

helm test — pods annotated helm.sh/hook: test, run on demand via helm test <release>:

• Smoke tests as shipped artifacts: a pod that curls the service through the Service DNS (testing the whole chain: service → endpoints → pod → app), checks DB connectivity with the chart's rendered credentials, validates the ingress answers externally if reachable
• Where they earn their keep: post-install verification in CI pipelines (helm upgrade -i --wait && helm test) — a deploy isn't done because pods are Ready; it's done because the test pod got a 200. Also golden for chart CI (ct install runs them) and for operators at 3am validating an env without knowing the app ('run helm test, if green the platform layer is fine')
• Discipline: tests must be hermetic (no external dependencies that flake), fast (<1 min), and cleaned up (hook-delete-policy: hook-succeeded — keep failures for debugging)

One-liner: 'NOTES.txt is your chart's onboarding doc computed per-install; helm test is its health contract — both turn tribal knowledge into shipped artifacts.'

The philosophical difference: Helm is templating + packaging + lifecycle (parameterize YAML with a values API, distribute versioned artifacts, track releases). Kustomize is patching (declare a base, overlay environment-specific patches — no templates, no variables, pure YAML transformation, built into kubectl).

Where Helm wins:

• Distribution: versioned artifacts, repos/OCI, dependencies — you can't kustomize install postgresql; third-party software ships as charts
• A values API for consumers — 30 teams configuring a paved-road chart via documented knobs beats 30 teams writing patches against your base's internals (patches couple to structure; values couple to contract)
• Conditional resources, loops, computed names — logic that patching fundamentally can't express

Where Kustomize wins:

• No template tax: what you read is YAML, not Go templates around YAML — diffs are honest, editors validate, no whitespace/nindent bugs
• Patching things you don't own: overlay changes onto any manifests without upstream cooperation — no waiting for a chart to expose a knob
• Env overlays for your own apps are often simpler: base + 10-line prod patch vs a values plumbing chain
• Zero lifecycle machinery to operate (also its weakness: no releases, no history, no rollback primitive — git is your only history)

The hybrid patterns (production-standard, not compromise):

1. Helm render → Kustomize patch: consume third-party charts, apply org-invariants (labels, registries, sidecars) as overlays — via Helm post-renderer or ArgoCD's multi-source apps. Upstream stays unforked; your delta is explicit
2. Charts for products, Kustomize for instances: platform ships charts (versioned, tested, values API); a team's env-specific tail (an extra ConfigMap, a prod-only NetworkPolicy) lives as overlay resources beside the Application — no chart change needed for one team's one-off
3. ArgoCD as the equalizer: Applications don't care — helm, kustomize, or both per app; lifecycle (sync, history, rollback) comes from Argo uniformly, which neutralizes Helm's lifecycle advantage and lets each app pick the right authoring tool

One-liner: 'Helm is how software is shipped to Kubernetes; Kustomize is how environments adapt it. Mature platforms use Helm at the distribution boundary and patching at the ownership boundary — and let GitOps own lifecycle for both.'

Three reuse mechanisms, constantly confused:

• Library chart (type: library in Chart.yaml): ships named templates only — it renders nothing by itself and can't be installed. Consumers declare it as a dependency and include its templates. It's a function library: corp.deployment, corp.labels, corp.hpa
• Umbrella chart: a regular chart whose value is its dependency list — composes N installable subcharts into one release. It's aggregation, not reuse of logic
• Starter (helm create --starter): a scaffold copied at chart-creation time — after generation there's no link; fixes to the starter don't propagate. It's a template in the copy-paste sense

Why library beats starter for platform standards: the starter gives teams a good day-1 chart that drifts forever after; the library chart gives them a dependency they version-bump — fix the PDB logic once in corp-lib 1.4.1, and every consuming chart picks it up on their next dependency update. Reuse with an upgrade path vs reuse as a photocopier.

The platform inheritance design:

corp-lib (library)          # named templates: workloads, labels, security, PDB, HPA
  ↑ dependency
service-chart (installable) # the paved road: 'a standard corp service'
  ↑ teams either use directly with values...
team charts (thin)          # ...or build on corp-lib for special shapes

1. corp-lib holds all opinion: security contexts, label taxonomy, probe defaults, topology spread — exposed as templates taking explicit dicts (testable, documented inputs)
2. service-chart is the 90% path: teams install it with values only, never write templates. It's itself a thin consumer of corp-lib
3. Teams with genuinely special shapes (the Kafka-consumer team, the GPU team) write thin charts on corp-lib directly — they inherit standards without inheriting the service-chart's assumptions
4. Versioning discipline doubles: corp-lib template outputs are API (golden-render tests across all consumers in CI); breaking output changes are majors with deprecation windows — a library chart's blast radius is every chart in the org

Umbrella's place in this: none for standards — only for shipping genuinely coupled component sets (an operator + CRDs + dashboards) as one versioned unit.

One-liner: 'starters copy, libraries link, umbrellas bundle — platform standards need the link, because standards without an upgrade path are just suggestions with a deadline.'

When it's TRUE — Helm's real modeling limits: Helm is a deploy-time tool: render, apply, exit. It has no runtime presence, so it fundamentally can't do:

• Reactive operations — failover on primary death, rebalancing on node loss, scaling on queue depth: someone must be watching; Helm isn't there
• Ordered stateful choreography — 'upgrade replicas one at a time, waiting for ISR/re-sync between each, leader last': Helm applies manifests; it can't sequence within the rollout beyond what StatefulSet semantics give
• State-dependent decisions — 'only compact if disk >70%', 'restore this replica from the last snapshot': requires reading live state and acting, continuously
• Anything where the correct next action depends on runtime state — that's a control loop, i.e. an operator

When it's an EXCUSE:

• The 'day-2 ops' are actually deploy-time config changes (resize resources, change flags, add replicas) — that's helm upgrade, and building an operator to avoid learning Helm is résumé-driven engineering
• One instance, quarterly manual maintenance — a runbook + Job beats a custom controller you must now version, test, secure, and staff
• The real complaint is 'our upgrade has 4 manual steps' — hooks, init-container gates, or a pipeline can encode 4 steps; an operator is for unbounded steps decided at runtime

The migration path (Helm chart → operator) without a big bang:

1. Keep Helm as the delivery vehicle — the operator itself ships as a chart (they compose: Helm installs the operator + CRDs; the operator manages the workload)
2. Strangler pattern on responsibilities: v1 operator handles only the hardest runtime concern (say, coordinated upgrades) while the chart still renders the base resources; each release moves a responsibility across as CRD fields replace values keys
3. CRD design mirrors the values API you already validated with users — your values schema is a first draft of your CRD schema; keep names consistent so migration is mechanical
4. Adoption mechanics: operator labels/adopts chart-created resources (the ownership-metadata dance), one workload at a time, with rollback = disable reconciliation + revert to chart-managed
5. Exit criteria honesty: if after v1 the operator is only templating (no reconcile logic beyond create-if-missing), stop — you've built a slow Helm; fold back

One-liner: 'Helm answers what should exist; operators answer what should happen next. Migrate when the second question is real and continuous — and even then, Helm still delivers the operator.'

What you're seeing: a rolling update mid-flight or wedged — both ReplicaSets scaled up, Service endpoints spanning both versions. Normal transiently; pathological when it persists or when the versions are wire-incompatible.

Why it persists — the usual causes:

1. New pods never go Ready (bad readiness, missing config/secret, image pull) → Deployment respects maxUnavailable and won't kill old pods → stable hybrid. Helm without --wait reported success the moment manifests applied — the pipeline is green, production is bi-versioned
2. Progress deadline hit: rollout gave up (ProgressDeadlineExceeded), stuck at whatever ratio it reached — check kubectl rollout status / Deployment conditions
3. Selector/labels drift: the upgrade changed pod labels but not the selector (or a second Deployment/old ReplicaSet still matches the Service selector) — the Service is selecting across two workloads; kubectl get endpoints + get pods -l <selector> --show-labels exposes it
4. PDB + node ops collision: a concurrent drain blocked by PDB while surge pods came up elsewhere — churn, not progress

Why sessions break: version skew is only safe if you engineered it — incompatible session serialization, cache formats, API contracts, or sticky-session dependence make v1↔v2 coexistence corrupting. Rolling updates guarantee a skew window; 'it broke during deploys' means the app violated that contract, not that Kubernetes misbehaved.

Immediate response: decide direction, don't linger — kubectl rollout undo (or helm rollback) if new is bad; fix-forward the readiness blocker if new is good. Confirm one ReplicaSet at full count and endpoints homogeneous.

Prevention stack:

• Pipeline honesty: --wait --timeout + --atomic — Helm success must mean 'rolled out', not 'submitted'; alert on Deployment Progressing=False
• Skew engineering: N/N-1 compatibility as a testing requirement (contract tests run old-client-vs-new-server and inverse), sessions externalized, serialization versioned
• When skew is truly intolerable: don't roll — blue-green with atomic traffic switch (two full stacks, flip at the Service/ingress), or strategy: Recreate where a downtime blip beats bi-versioning
• Readiness gates that reflect real serveability, so 'Ready' can't lie the rollout forward

What they're testing: you read hybrid state as a stuck state machine (find which guard failed), you know Helm's success ≠ rollout success without --wait, and you treat version-skew compatibility as an application contract, not luck.