Kubernetes

Kubernetes is a container orchestrator: you declare the desired state of your workloads (what runs, how many copies, how they're exposed) and its control loops continuously reconcile reality toward that declaration.

The problems it solves:

• Scheduling — bin-packing containers onto a fleet of machines
• Self-healing — dead containers restart, dead nodes get their pods rescheduled
• Scaling — horizontal scaling of replicas, and of nodes (with an autoscaler)
• Service discovery & load balancing — stable virtual IPs/DNS in front of ephemeral pods
• Rolling deployment — versioned, controlled rollout and rollback as a primitive

What interviewers listen for: the phrase declarative + reconciliation loop — Kubernetes isn't a deploy script, it's a control system that never stops correcting drift.

• kube-apiserver — the front door; the only component anything talks to. Validates, authenticates, authorizes, persists to etcd. Stateless, horizontally scalable.
• etcd — the source of truth: a Raft-based consistent key-value store holding all cluster state. Quorum-based (3 or 5 nodes); lose quorum, the cluster is read-only-ish frozen.
• kube-scheduler — watches for unscheduled pods, scores nodes (resources, affinity, taints, spread), binds pod → node. It only decides; kubelet does the running.
• kube-controller-manager — dozens of control loops (Deployment, ReplicaSet, Node, Job, EndpointSlice...) each reconciling actual → desired state.
• cloud-controller-manager — cloud glue: provisions LBs, attaches volumes, syncs node lifecycle.

On workers: kubelet (runs pods via the CRI runtime — containerd), kube-proxy (Service routing via iptables/IPVS), the container runtime.

Senior detail: everything communicates through the API server watching for changes — components never talk to each other directly. That watch-based, level-triggered design is why the system tolerates component restarts.

A Pod is one or more containers that share a network namespace (same IP, talk over localhost), shared volumes, and a lifecycle — always scheduled together on the same node.

Why pods, not containers: some workloads are genuinely co-located units — an app plus a log shipper, a service plus a proxy sidecar (the service-mesh pattern), an init container preparing state. They need shared fate, shared network, shared disk. The pod formalizes that.

Key properties:

• Ephemeral by design — pods are never healed, they're replaced. Controllers (Deployments) manage replacement.
• Each pod gets a unique cluster IP — no port juggling between apps on one host.
• Init containers run to completion before app containers start; sidecars run alongside (native sidecar support via restartPolicy Always on init containers since 1.28+).

Trap to avoid: putting your app and its database in one pod 'so they're together' — pods scale and die as one unit; that couples scaling of two things that must scale independently.

• Deployment — stateless workloads; the default. Manages ReplicaSets for rolling updates/rollbacks. Pods are interchangeable cattle: random names, any node, shared storage nothing.
• StatefulSet — when pods need stable identity: ordered names (db-0, db-1), stable DNS per pod, and per-pod persistent volumes that survive rescheduling. For databases, Kafka, anything where 'which replica am I' matters. Rolling updates proceed in reverse ordinal order.
• DaemonSet — exactly one pod per node (or per selected nodes): log collectors, node monitoring agents, CNI/CSI plugins, security agents. New node joins → pod appears automatically.
• Job — run-to-completion tasks with retries and parallelism controls; CronJob schedules them. Batch processing, migrations, report generation.

Interview follow-up to expect: 'should you run databases in StatefulSets?' Honest answer: you can, with operator maturity (e.g. a Postgres operator handling failover/backup), but managed database services usually win on TCO unless you have platform-team depth or specific requirements.

A Service gives a stable virtual IP + DNS name to an ever-changing set of pods, selected by labels. The EndpointSlice controller tracks ready pod IPs; kube-proxy programs each node (iptables or IPVS rules) so traffic to the service IP gets DNATed to a random ready pod.

Types (each builds on the previous):

• ClusterIP — internal-only virtual IP; the default; service-to-service traffic.
• NodePort — additionally opens a port (30000-32767) on every node forwarding to the service. Rarely used directly in production — it's the building block LBs target.
• LoadBalancer — additionally provisions a cloud load balancer (via the cloud controller) pointing at the nodes/pods. One LB per service — costly at scale, which is why Ingress/Gateway API exists to multiplex.
• Headless (clusterIP: None) — no VIP; DNS returns the pod IPs directly. Used by StatefulSets and client-side load-balancing (e.g. gRPC).

Senior details: the service IP is virtual — it never appears on any interface, it exists only as dataplane rules; and externalTrafficPolicy: Local preserves client source IPs at the cost of node-local-only routing.

A Service exposes one workload at L4. An Ingress is L7 HTTP(S) routing config — host- and path-based rules mapping many hostnames/paths to many services through one entry point:

• api.example.com/v1/* → api-service
• app.example.com → frontend-service
• TLS termination per host (cert references in the spec)

Critically, Ingress is just an API object — it does nothing without an ingress controller (NGINX, HAProxy, Traefik, or cloud-native like AWS Load Balancer Controller translating rules into ALB config). The controller watches Ingress resources and configures an actual proxy.

Economics: one LB + one controller fans out to hundreds of services vs one cloud LB per LoadBalancer service.

Where this is heading (say this): Gateway API is the successor — role-separated (infra owns Gateways, app teams own HTTPRoutes), first-class support for traffic splitting, header matching, gRPC — solving Ingress's annotation-sprawl problem where every controller invented its own annotations.

Requests = what the scheduler reserves — a node must have that much unallocated capacity for the pod to schedule. Limits = the runtime ceiling.

Behavior differs by resource:

• CPU is compressible: exceeding the limit → throttling (CFS quota), not death. Latency degrades; the container keeps running.
• Memory is incompressible: exceeding the limit → OOMKill of the container (exit 137, restart per policy).

QoS classes (derived, not chosen): requests == limits for all resources → Guaranteed; some requests set → Burstable; none → BestEffort. Under node memory pressure, eviction order is BestEffort → Burstable-over-request → Guaranteed last.

Production stance worth stating:

• Always set memory request = memory limit (predictable OOM behavior, no node-pressure surprises)
• Set CPU requests honestly (scheduling accuracy); many shops omit CPU limits to avoid needless throttling — know the debate and that throttling shows up as p99 latency with low CPU usage (container_cpu_cfs_throttled_periods_total)
• Requests drive cost — over-requesting is invisible waste (cluster at 30% real utilization but 'full' by requests is the classic pattern)

• Readiness — 'can I take traffic?' Fail → pod removed from Service endpoints. Recoverable condition (warming cache, waiting on dependency).
• Liveness — 'am I alive at all?' Fail → container restarted. For unrecoverable states only (deadlock, wedged event loop).
• Startup — 'still booting.' Disables the other two until it passes; for slow-starting apps (legacy JVMs) so liveness doesn't kill them mid-boot.

Classic misconfigurations (interviewers love these):

1. Liveness probe checks a dependency (DB, downstream API) → dependency blips, every pod restarts simultaneously, turning a degradation into a full outage. Liveness must check only the process itself.
2. Liveness == readiness endpoint — restarts where removal-from-rotation was the right response.
3. Timeouts too tight — probe timeout 1s on a service with occasional 2s GC pauses = restart storms under load, exactly when you can least afford them.
4. No startup probe on a slow app, so initialDelaySeconds is guessed and wrong on cold nodes.

Rule that lands: readiness = 'me + my ability to serve'; liveness = 'me only, conservative, with generous thresholds.'

A namespace is a logical partition of a cluster: a scope for names (same Deployment name can exist in team-a and team-b), for RBAC (roles binding users to permissions within a namespace), for ResourceQuotas (CPU/memory/object-count budgets per team), and for LimitRanges (default requests/limits injected per pod).

What namespaces are good for:

• Multi-team soft multi-tenancy — team-per-namespace with quota + RBAC + NetworkPolicy is the standard shared-cluster pattern
• Environment separation within reason (dev/staging in one cluster; prod usually merits its own cluster)
• Blast-radius scoping for operators and controllers

What they are NOT: a security boundary by themselves. Namespaces don't isolate the network (all pods can talk cross-namespace until you add NetworkPolicies), don't isolate nodes (pods from different namespaces share kernels), and cluster-scoped resources (nodes, CRDs, ClusterRoles) ignore them entirely.

Senior line: 'namespace = administrative boundary; node/cluster = security boundary. Hostile tenants need separate clusters or hard multi-tenancy machinery, not just namespaces.'

Both decouple config from images; both inject the same ways:

• Env vars — simple, but frozen at container start; changes require a restart, and env vars leak into crash dumps/child processes
• Volume mounts — files in the container; ConfigMap/Secret updates propagate to mounted files (within ~a minute, via kubelet sync) if the app re-reads them; subPath mounts do NOT update
• Referenced by other objects (image pull secrets, TLS in Ingress)

How secret is a Secret? Barely, by default:

• Values are base64-encoded, not encrypted — encoding ≠ encryption
• Stored in etcd; without encryption at rest configured (KMS provider — default on EKS since 2025-era, but verify), anyone with etcd access reads everything
• Anyone with get secrets RBAC in the namespace reads them; pods mounting them can read them; node compromise exposes secrets of pods on that node

Production hardening (the expected answer): enable etcd encryption via KMS, RBAC least-privilege on secrets (no wildcard get secrets), and prefer an external secret store — External Secrets Operator or Secrets Store CSI driver pulling from Vault/AWS Secrets Manager — so rotation and audit live in a purpose-built system and the cluster holds only short-lived material.

The canonical depth-probe. Hit the whole chain:

1. kubectl — client-side validation, builds the request, authenticates (kubeconfig creds) to the API server.
2. API server — authentication → authorization (RBAC: can this identity create Deployments here?) → admission chain: mutating webhooks (sidecar injectors, defaulters) → schema validation → validating webhooks/policies (OPA/Kyverno). Only then is the object persisted to etcd (quorum write).
3. Deployment controller (in controller-manager, watching via the API server) sees a new Deployment → creates a ReplicaSet.
4. ReplicaSet controller sees desired replicas=N, actual=0 → creates N Pod objects (still unscheduled).
5. Scheduler watches for pods with no nodeName: filters nodes (resources, taints/tolerations, affinity, topology spread) → scores survivors → writes a binding (pod.nodeName = chosen node).
6. kubelet on that node (watching pods bound to it): pulls images, asks the CRI runtime (containerd) to create sandbox + containers, invokes the CNI plugin to wire the pod network (IP allocation, routes), mounts volumes via CSI, runs init containers, starts app containers, executes probes.
7. Readiness passes → pod condition Ready → the EndpointSlice controller adds the pod IP to the Service's endpoints → kube-proxy on every node updates iptables/IPVS rules → traffic flows.

What separates senior answers: naming the watch mechanism (every step is a controller reacting to API-server watch events — no orchestrator calls anyone), the admission chain's position before etcd, and that scheduling and running are decoupled (scheduler decides, kubelet executes).

What etcd needs: Raft quorum — (n/2)+1 members. 3 nodes tolerate 1 loss; 5 tolerate 2. Even counts add risk without adding tolerance.

Failure modes in practice:

1. Quorum loss — API writes fail cluster-wide; workloads keep running (kubelets and the dataplane don't need etcd moment-to-moment) but nothing reconciles: no new pods, no failover, no scaling. The cluster is a photograph.
2. Disk latency — etcd is fsync-bound; slow disks (>10ms WAL fsync) cause leader-election churn and API timeouts that look like 'Kubernetes is flaky'. NVMe/io2 storage, dedicated disks. Watch etcd_disk_wal_fsync_duration_seconds (p99 < 10ms) and etcd_server_leader_changes_seen_total.
3. DB size bloat — default quota 2GB (max ~8GB); hit it → cluster goes read-only with a NOSPACE alarm. Causes: event spam, huge objects, resource churn without compaction. Fix: compaction + defrag (defrag is stop-the-world per member — roll it), and stop writing garbage (event TTLs, don't stuff data in annotations).
4. Split brain risk in stretch topologies — 2-zone control planes are worse than they look (either zone loss can kill quorum); 3 zones for control planes always.

Operations checklist: scheduled etcdctl snapshot save + restore actually rehearsed (an unrestored backup is a hope), TLS peer/client, version marching with the control plane, and network isolation (etcd compromise = cluster compromise, and secrets live there).

Managed-control-plane note: on EKS/GKE this is AWS/Google's problem — but the symptoms (API latency, throttling, object bloat you cause) are still yours to diagnose.

The model (three rules): every pod gets a unique IP; all pods reach all pods without NAT; a node reaches its own pods without NAT. How that's implemented is delegated entirely to the CNI plugin.

What the CNI does at pod creation: kubelet → CRI → CNI plugin: allocate an IP (IPAM), create a veth pair (one end in the pod's netns as eth0, the other on the node), wire the node side (bridge, or direct routes), program routes/encapsulation so other nodes can reach this IP.

Pod A (node 1) → pod B (node 2), by CNI flavor:

• Overlay (VXLAN — Flannel, Cilium default): packet leaves pod A's veth → node routing decides pod B's IP is remote → encapsulated in VXLAN (UDP 8472) addressed node1→node2 → node 2 decapsulates → local route → pod B's veth. Cost: encap overhead + MTU reduction (watch for mystery fragmentation).
• Native routing (Calico BGP, AWS VPC CNI): pod IPs are routable in the underlying network — no encapsulation. VPC CNI goes further: pod IPs are real VPC IPs on ENIs, so the VPC route table itself delivers the packet. Cleanest dataplane; the cost is IP consumption (the EKS IP-exhaustion class of problems).
• eBPF dataplanes (Cilium): replace iptables/kube-proxy entirely — service resolution and policy in eBPF at the socket/XDP layer; big rule-count scalability and observability (Hubble) wins.

Where people get bitten: MTU mismatches with overlays, conntrack table exhaustion on high-connection nodes, source-IP loss through SNAT (externalTrafficPolicy), and assuming namespaces isolate the network (they don't — that's NetworkPolicy's job).

Senior close: 'the CNI choice is a real architecture decision — IP consumption vs encap overhead vs policy/observability features — not a default you inherit.'

The mechanism: kube-proxy on every node watches Services + EndpointSlices and programs the node's dataplane so connections to a ClusterIP get DNATed to a backend pod IP. The service IP is pure fiction — rules, not an interface.

Modes:

• iptables (default): rule chains per service with probabilistic DNAT for balancing. O(n) rule evaluation and full-table updates; at ~5-10K+ services, rule-sync latency and packet-path cost degrade — service churn takes seconds-minutes to propagate.
• IPVS: kernel L4 load balancer — hash-table lookups O(1), real algorithms (rr, least-conn), dramatically better at scale. Still uses iptables for a few functions (SNAT marks).
• eBPF (Cilium kube-proxy replacement): socket-level LB — for pod-originated traffic the connect() is rewritten to the backend directly, skipping the whole per-packet NAT path.

Sharp edges worth naming:

1. conntrack — every flow through the service path occupies a conntrack entry; high-connection-rate nodes exhaust nf_conntrack_max → dropped SYNs that look like app flakiness. A classic UDP variant: stale conntrack entries after backend churn black-hole DNS until timeout.
2. externalTrafficPolicy: Cluster (default) SNATs and may double-hop but spreads load; Local preserves client IP and avoids the extra hop but only routes to node-local pods — imbalance if pods are unevenly spread, and health checks must reflect local presence.
3. Session affinity is ClientIP-only and coarse; real stickiness belongs at L7.
4. Long-lived connections defeat rebalancing — gRPC clients pinned to one backend via a service VIP need client-side LB (headless service) or a mesh.

What they're testing: you know the VIP is dataplane rules, you can name the scale ceiling of iptables mode, and you've debugged conntrack at least once.

Principles: identity from your IdP (OIDC), permissions to groups never users, namespace-scoped by default, cluster-scoped grants treated like production changes.

The design:

1. Identity: OIDC via the IdP (Okta/Entra); group claims map team membership. No static ServiceAccount tokens for humans, ever.
2. Standard role tiers per namespace (defined once as ClusterRoles, bound per-namespace via RoleBindings):
   • team-viewer — get/list/watch on workloads + logs
   • team-developer — viewer + create/update workloads, port-forward; no secrets read, no RBAC edit
   • team-admin — developer + secrets, quota visibility, rolebinding within approved role set
3. Platform roles: cluster-admin held by ~3 people via break-glass (JIT elevation, audited, time-boxed) — daily platform work uses a scoped platform-operator role.
4. Workload identity: one ServiceAccount per app, automountServiceAccountToken: false unless the app talks to the API.

Shadow-admin escalation paths (the part that separates seniors):

• create pods + node access = read any secret mountable in that namespace; pods/exec into a pod with a powerful SA = you are that SA
• escalate/bind/impersonate verbs — grant bind carelessly and users self-promote by binding existing powerful roles
• Write access to mutating webhooks, or to controllers/operators' CRDs that act with high privilege = privilege laundering through the controller
• update deployments in a namespace containing a high-priv SA = swap the pod spec to use it

Guardrails: Kyverno/OPA policies denying dangerous grants, periodic access review from an RBAC graph tool (who effectively has cluster-admin — the answer always surprises people), audit logs on secrets access and exec.

Closing line: 'RBAC review isn't reading Roles — it's computing reachable privilege through pods, SAs, and controllers.'

Structured triage, narrating the why:

1. kubectl describe pod <pending> — Events tell you 90% of it. The usual suspects:
   • Insufficient cpu/memory → cluster capacity vs requests
   • didn't match node selector/affinity → label mismatch (nodepool renamed?)
   • had untolerated taint → new node pool taints, or all nodes tainted (e.g. NotReady)
   • volume node affinity conflict → PV pinned to AZ-a, schedulable capacity only in AZ-b — the classic StatefulSet zone trap
   • exceeded quota → namespace ResourceQuota exhausted (rollout surge needs headroom above steady state)
2. Capacity path: kubectl top nodes + check if cluster autoscaler/Karpenter is trying: its logs/events say why scale-up isn't happening (max-size reached, instance type unavailable, ASG quota, spot exhaustion).
3. Rollout math: maxSurge/maxUnavailable vs quota and capacity — surge=25% on a 40-replica deployment needs 10 spare pods' worth of room. PodDisruptionBudgets don't block rollouts, but a too-strict PDB plus a concurrent node drain absolutely wedges things — check for in-flight node operations.
4. If pods schedule but never Ready (different flavor of stuck): image pull (private registry auth, rate limits), failing readiness probe (new version's dependency broken), or admission webhook latency.
5. Mitigate vs fix: if capacity, scale the node group / relax surge; if config, kubectl rollout undo restores service while you root-cause — say the rollback option out loud; interviewers check whether you restore service before debugging to perfection.

What they're testing: Events-first discipline, knowing scheduling constraints compose (ALL must pass), and the quota/surge interaction that bites real teams.

Three loops at different layers:

• HPA — scales replicas on metrics: CPU/memory utilization (vs requests — key detail), custom metrics (RPS, queue depth via Prometheus adapter), or external metrics (SQS length via KEDA). Control loop every 15s; scale-down stabilization window (default 5min) prevents flapping.
• VPA — adjusts requests/limits per pod from observed usage. Modes: Off (recommendations only — the safe, popular mode), Initial, Auto (historically required pod restarts; in-place resize is maturing in recent versions).
• Cluster Autoscaler / Karpenter — scales nodes: CA grows/shrinks predefined node groups when pods are unschedulable; Karpenter provisions right-sized instances directly from pending-pod requirements (faster, better bin-packing, consolidation for cost).

The interaction bugs (the real question):

1. HPA + VPA on the same metric = fight. VPA raises requests → utilization % (usage/requests) drops → HPA scales in → load concentrates → VPA raises again. Never both on CPU/memory for one workload; HPA-on-custom-metric + VPA is the safe combination.
2. HPA scales on utilization-vs-requests — wrong requests make HPA nonsense: requests too low → HPA over-scales; too high → never triggers until users hurt.
3. HPA wants pods, cluster can't grow — max nodes reached, instance unavailability, quota → pods Pending while HPA keeps raising desired. Alert on pending pods age, not just HPA status.
4. Scale-down cascade: aggressive CA consolidation + tight PDBs = eviction churn; Karpenter consolidation during business hours needs do-not-disrupt annotations on sensitive workloads.
5. Metric lag: scaling on p99 latency (a trailing indicator) oscillates; scale on leading signals (RPS, queue depth) and set sane stabilization windows.

Senior stance: requests are the contract everything reads — get them right (VPA in recommend mode feeding the values), scale replicas on a leading business metric, and let Karpenter handle machines.

Semantics you must state precisely: policies are allow-only and additive; selecting a pod flips it from allow-all to default-deny for the selected direction(s). No policy selecting a pod = everything allowed. Enforcement is the CNI's job — a cluster without a policy-capable CNI silently ignores all of them (the worst failure mode: imagined security).

The pattern for PCI:

1. Default-deny baseline per namespace — empty-podSelector policy denying all ingress and egress. Everything after is explicit allowlisting.
2. DNS first (everyone forgets): allow egress UDP/TCP 53 to kube-dns, or every connection breaks at name resolution.
3. Tiered flows: payment-api accepts ingress only from the gateway namespace (namespaceSelector + podSelector); DB namespace accepts 5432 only from payment-api pods; egress from PCI namespaces enumerated to specific dependencies — including CIDR blocks for managed services (RDS endpoints).
4. Cross-cutting allowances: observability scrape ingress, egress to the mesh control plane if applicable — defined once as standard policy fragments stamped into every namespace by the platform (Kyverno generate).

Gaps and their fixes:

• Vanilla policies are L3/L4 only — no FQDN egress rules ('allow api.stripe.com') and no L7; Cilium/Calico Enterprise extensions add FQDN and L7 policy, or push egress through an inspecting proxy
• No default cluster-wide deny primitive — namespace coverage must be enforced by policy-as-code (audit for namespaces lacking a default-deny)
• Policies don't apply to hostNetwork pods, and the node itself is outside the model — node-level firewalls/SGs still matter
• Auditors want evidence: flow logs (Hubble/Calico) proving denied traffic, not just YAML

Senior close: 'policy objects are the easy half; the hard half is CI validation that new services ship with policies, and flow observability to prove the segmentation is real.'

PSP was removed in 1.25, replaced by Pod Security Admission (PSA) — namespace labels enforcing three profiles: privileged (no restrictions), baseline (blocks known escalations: hostNetwork, privileged, hostPath), restricted (hardened: non-root, seccomp, dropped capabilities). Each in enforce/audit/warn modes — roll out with audit first, then enforce. For anything conditional or bespoke (allowed registries, required labels, exceptions), Kyverno or OPA Gatekeeper does what PSA deliberately doesn't.

A hardened spec, annotated:

securityContext:            # pod-level
  runAsNonRoot: true
  runAsUser: 10001
  fsGroup: 10001
  seccompProfile: { type: RuntimeDefault }   # syscall filter baseline
containers:
- securityContext:
    allowPrivilegeEscalation: false          # blocks setuid/sudo paths
    readOnlyRootFilesystem: true             # writes only to declared volumes
    capabilities: { drop: ["ALL"] }          # add back only what's proven needed

Plus around the spec: automountServiceAccountToken: false unless the app calls the API; no hostPath/hostNetwork/hostPID; image by digest from a private registry, minimal/distroless base; resource limits set (a fork bomb is a resource attack).

Why each control matters (one-liners): non-root + no-escalation kills most container-escape preconditions; read-only rootfs blocks payload-drop persistence; RuntimeDefault seccomp filters ~40+ risky syscalls; dropped capabilities remove CAP_NET_RAW spoofing and friends.

The escalation ladder to name: privileged pod ≈ root on node → node kubelet creds → secrets of every pod on the node → often cluster-admin via a mounted SA. That ladder is why 'just one privileged pod' in a shared cluster is a cluster-level risk.

Senior close: enforcement lives in admission (PSA + policy engine), verification in CI (scan manifests), and exceptions are namespaced, documented, and expiring.

Ground rules: minor versions upgrade one at a time (1.29→1.30→1.31, no skipping); control plane first, nodes after; version skew allows kubelet up to n-3 behind the API server — that skew window is what makes rolling node upgrades safe.

Phase 0 — pre-flight (where upgrades are actually won):

• API deprecation sweep: kubectl-convert/Pluto/kube-no-trouble against manifests and Helm charts, not just live objects — removed APIs (the recurring Ingress/PSP-style breakages) must be migrated before, not during
• Add-on compatibility matrix: CNI, CSI, ingress controller, mesh, cert-manager, operators — each pinned to a version supporting both K8s versions in flight
• Webhook inventory: a failing admission webhook during upgrade can block all pod creation — know your failurePolicy settings
• Rehearse the entire sequence in a staging cluster built like prod (same add-ons, same webhooks), including a workload soak

Phase 1 — control plane: managed (EKS) = API server upgrades in place, brief connection blips only — clients need retry logic. Self-managed: one control-plane node at a time behind the LB.

Phase 2 — nodes, surge-style: create new-version node group (or let Karpenter drift-replace) → cordon old nodes → drain respecting PDBs with parallelism tuned to capacity headroom → verify workloads on new nodes → delete old group. Requirements that make this safe: every critical workload has PDBs (else drain is an outage), topology spread across nodes/AZs, graceful termination handled (preStop hooks, connection draining), and stateful workloads reviewed individually.

Phase 3 — add-ons + validation: upgrade add-ons in their supported order, run conformance/smoke suites, watch error budgets for a soak period before declaring done. Then immediately do it again for the second minor.

Rollback honesty: control planes don't downgrade — your rollback is node-group revert + restore from backup (Velero) for worst case, which is why staging rehearsal and one-minor-at-a-time discipline matter.

Cadence point: K8s ships ~3 releases/year with ~14-month support — upgrading is a program, not an event; clusters more than 2 versions behind are risk debt compounding.

The plumbing: CoreDNS runs as a Deployment behind the kube-dns ClusterIP; every pod's /etc/resolv.conf points there. Records: svc-name.namespace.svc.cluster.local (service VIP), pod-level and SRV records for headless services. The ndots:5 default means any name with <5 dots gets search-path expansion: api.prod is tried as api.prod.default.svc.cluster.local., api.prod.svc.cluster.local., etc., before the literal name — multiplying queries per lookup (external names generate 4-5x query load).

The 5-second mystery (a real classic): intermittent exactly-5s delays = DNS UDP packet loss + the default 5s timeout, historically triggered by a conntrack race on parallel A+AAAA queries from the same socket (same 5-tuple, insertion race drops one packet). glibc waits 5s and retries.

Debug path:

1. Confirm shape: latency histogram shows a spike at exactly 5.0s → timeout+retry, not slow DNS
2. Check conntrack: conntrack -S insert_failed counter climbing on affected nodes
3. Check CoreDNS itself: CPU throttling (it's often under-resourced), error/latency metrics, upstream resolver health for external names

Fixes, in order of preference:

• NodeLocal DNSCache — DaemonSet cache on every node; pods query the local cache (no conntrack race, no cross-node hop); the standard production answer
• dnsConfig options: single-request-reopen / use-vc (TCP) workarounds for the race
• Tune ndots down (or use FQDNs with trailing dots in app config) to kill search-path amplification
• Right-size CoreDNS (HPA on it), cache tuning, and autopath plugin for the expansion problem

Senior signal: you recognize 'exactly 5 seconds, intermittent' as a signature, not a mystery — and you mention that DNS is the highest-QPS service in most clusters and deserves capacity planning like one.

CRD = extend the API with your own resource types (kind: PostgresCluster); you get storage, RBAC, kubectl, watch semantics for free. Operator = a controller reconciling those custom resources into real state — encoding operational knowledge (provisioning, failover, backup, upgrade) as software running a control loop.

The pattern's power: the same level-triggered reconciliation Kubernetes uses everywhere — observe desired (spec), observe actual (status), converge, repeat. Mature examples: Prometheus operator, cert-manager, ArgoCD, CloudNativePG, Strimzi.

Build one when:

• You operate many instances of a stateful/complex system and the runbook is genuinely automatable (the operator replaces a human following steps)
• You're a platform team exposing a product-like abstraction (kind: TenantDatabase) with your org's policies baked in
• Day-2 operations (failover, resize, upgrade) — not just day-1 install — are the pain

It's a mistake when:

• Helm chart + pipeline already solves it — install/configure is not an operator use case; reconciliation must earn its complexity
• One instance, rarely changed — you're building a distributed system (leader election, idempotent reconcile, status conditions, upgrade compatibility for the CRD schema itself) to avoid a quarterly manual task
• The team can't staff maintaining it — an unowned operator with cluster-wide write access is both an ops and a security liability
• A managed service exists — RDS beats your Postgres operator for most orgs

Engineering realities to name: idempotency (reconcile runs constantly, not once), status/conditions as the API contract, CRD versioning + conversion webhooks (schema migration is the hard part), and RBAC scoping so the operator isn't a shadow cluster-admin.

One-liner: 'operators are how you ship operations as software — justified exactly when the operations are complex, repeated, and yours.'

Situation: 600-node cluster, requests-based 'full', real CPU utilization 28% — paying for a phantom cluster. Teams over-request out of fear: nobody is rewarded for tight requests, everyone is paged for OOMKills.

Task: raise real utilization toward 55-60% without increasing incident rate — and make it stay fixed.

Action:

1. Measure and expose: per-namespace dashboards of requested-vs-used (P95 over 30 days) + a monthly 'slack cost' number in dollars per team. Visibility alone moved several teams.
2. VPA in recommendation mode everywhere — humans review suggested requests; no auto-apply on day one (trust first). Golden rule shipped as policy: memory request = P99 usage + headroom, memory limit = request; CPU request = P95, CPU limit generally unset (throttling causes the latency incidents people fear).
3. Renovate the incentive: namespace ResourceQuotas sized from measured need + growth, reviewed quarterly — over-requesting now visibly costs the team their own quota, not the platform's money.
4. Safety nets so tightening is survivable: PDBs everywhere, priority classes (critical workloads preempt batch), and Karpenter consolidation to actually harvest freed capacity into fewer nodes (requests fixed but nodes not consolidated = no savings).
5. Burstable batch tier: CI/batch moved to BestEffort/low-priority on spot capacity — they soak the slack instead of reserving peak.
6. Rollout discipline: one pilot org → two sprints of soak with error-budget watch → org-wide with exception path (teams can keep padded requests if they accept the itemized cost).

Result: utilization 28%→54% in a quarter, node count -38% (~$210K/yr on this cluster alone), OOMKill rate down (right-sized memory with limits==requests is more predictable than folklore numbers), and the requested-vs-used report is now a standing platform KPI.

The insight interviewers want: this is an incentive-design problem wearing a technical costume — tooling (VPA/Karpenter) is necessary but the fix is making waste visible and owned.

Ingress-NGINX (in-cluster proxy): LB → NGINX pods → upstream pods directly (bypassing kube-proxy). Rich behavior via annotations (rewrites, rate limiting, canary weights, auth subrequests), portable across clouds, huge install base. Costs: you operate it (scaling, tuning, CVEs — and it's a high-value attack target), config reloads at scale historically painful, and annotation sprawl is untyped, untestable API surface — the design flaw that motivated Gateway API.

Cloud-native (AWS Load Balancer Controller → ALB): ingress rules compile to ALB listeners/target groups; IP-target mode sends LB traffic straight to pod ENIs — no in-cluster hop, no proxy fleet to run. You inherit ALB features (WAF, ACM, Shield, OIDC auth) and give up NGINX-style programmability; per-Ingress-ALB defaults surprise people on cost (use IngressGroup to share one ALB).

Gateway API — the successor, GA and where investment is going:

• Role-oriented resources: infra team owns GatewayClass/Gateway (the LB, TLS, IPs); app teams own HTTPRoute attached to it — the multi-tenancy split Ingress never had
• Typed, portable semantics: traffic splitting (canary weights), header matching/manipulation, cross-namespace routing with explicit grants (ReferenceGrant) — as spec fields, not vendor annotations
• Implementations across the ecosystem (NGINX Gateway Fabric, Envoy Gateway, Cilium, cloud controllers), plus GAMMA extending it to east-west/mesh traffic — one API converging north-south and mesh routing

Selection heuristic: single-cloud, want managed dataplane → cloud controller (increasingly via Gateway API). Need L7 programmability or portability → Envoy-based Gateway API implementation; NGINX where it's already entrenched. Greenfield: start Gateway API — new routing features land there, Ingress is frozen.

Senior close: 'the direction is separation of concerns — platform owns Gateways like they own clusters; app teams self-serve Routes like they self-serve Deployments.'

The core insight: the OOM killer acts on the cgroup's full memory accounting, while the dashboard usually plots container working_set or the app's heap — several real consumers live in the gap:

1. Page cache & dirty pages — file-backed memory counts against the cgroup (mostly reclaimable, but dirty pages under heavy write flushing aren't instantly); log-heavy or file-churning apps spike here invisibly
2. Off-heap — the classic JVM case: heap capped at 512MB inside a 1GiB limit, but metaspace + thread stacks (1MB × threads) + direct byte buffers + JIT code cache + native libs push the process well past heap. Same story for Python C-extensions, Go cgo, native image libraries
3. memory spikes between scrapes — Prometheus samples every 30-60s; a burst allocation (big response buffering, batch job) OOMs in seconds and the graph never shows it — 'flat at 60%' is an artifact of sampling
4. emptyDir on tmpfs (medium: Memory) and shm — files there are memory charged to the pod
5. Sidecars share nothing — but check which container was killed: lastState.terminated.reason: OOMKilled per container; people debug the app while the log sidecar is the victim

Debug sequence:

• kubectl describe pod → which container, exit 137, restart pattern (every few hours = something periodic: cron in-app, cache TTL, batch window?)
• Node dmesg/journal for the OOM record — it dumps the cgroup's RSS/cache breakdown at kill time (ground truth)
• Compare container_memory_working_set_bytes vs container_memory_rss vs container_memory_cache; check tmpfs mounts
• For JVM: -XX:MaxRAMPercentage=75 (not a hardcoded heap), NativeMemoryTracking to inventory off-heap

Fixes: account for total-process memory in the limit (heap ≈ 60-70% of limit for JVMs), tune the runtime to respect cgroup limits (modern JVMs/dotnet do; check flags), avoid memory-backed emptyDir for large files, and alert on working_set/limit > 85% and container restart reason, not just averages.

What they're testing: you know limits police the cgroup, not the heap — and that monitoring granularity can hide the truth.

The spectrum and its trade-offs:

1. Shared cluster, namespace tenancy (soft multi-tenancy):

• Isolation stack required: RBAC per namespace + ResourceQuota/LimitRange + NetworkPolicy default-deny + PSA restricted + priority classes; optionally dedicated node pools (taints) for noisy/sensitive tenants
• Pros: best utilization (shared headroom), one platform to operate, fastest tenant onboarding (minutes)
• Cons: shared kernel and control plane — a container escape or an API-server-melting tenant affects everyone; CRD/operator versions are cluster-global (team A needs cert-manager v1.x, team B v2.x = conflict); blast radius of upgrades is total

2. Cluster per team (hard multi-tenancy):

• Pros: real isolation (security and failure), independent upgrade cadence, per-team CRD freedom, clean cost attribution
• Cons: fleet management is now the product (30 clusters = 30 upgrade cycles — needs Cluster API/Fleet/ArgoCD ApplicationSets and a management plane), utilization drops (per-cluster headroom), per-cluster fixed costs

3. Virtual clusters (vcluster) — the middle path: tenant gets its own API server + CRDs + versions, pods land on shared nodes. Solves control-plane and CRD isolation cheaply; does not solve kernel/node isolation. Excellent for dev/preview environments; maturing for prod.

My call for a typical 30-team org:

• Shared clusters by environment tier (a few prod cells, staging, dev) with the full soft-tenancy stack for most teams
• Dedicated clusters only by exception: regulated workloads (PCI/PHI), genuinely hostile isolation needs, or teams whose CRD/version requirements conflict with the fleet
• vclusters for ephemeral/dev — preview environments per PR without cluster sprawl
• Non-negotiable foundation: tenancy-as-code (onboarding = one PR generating namespace, quota, RBAC, policies, budgets) and per-tenant cost showback

Senior framing: 'tenancy model = blast-radius budget. I ask what must never share fate — kernel? control plane? upgrade window? — and buy exactly that much isolation, because every step up the spectrum trades utilization and operational leverage for it.'

Definition: a dedicated L7 infrastructure layer for service-to-service traffic — historically sidecar proxies (Envoy) next to every pod, programmed by a control plane (Istio, Linkerd); newer variants move the dataplane into the node/kernel (Istio ambient, Cilium mesh).

What it genuinely solves (in value order):

1. mTLS everywhere — workload identity (SPIFFE-style), automatic cert issuance/rotation, encrypted east-west traffic — the compliance answer for 'zero trust internal traffic' without touching app code
2. Uniform L7 telemetry — per-hop golden signals and distributed-trace propagation for every service, including the ones whose teams never instrumented anything
3. Traffic policy as config — retries with budgets, timeouts, outlier ejection, circuit breaking, canary weighting, fault injection — consistent across languages instead of per-app library roulette
4. AuthZ between services — 'payments accepts calls only from checkout' as enforced policy, not convention

The honest costs: a distributed system inside your distributed system — control-plane operations, version upgrades across hundreds of sidecars, per-pod resource overhead (50-100m CPU, 50-100MB each — real money at 5K pods), added p99 latency (0.5-2ms/hop), and debugging complexity (is it the app, the sidecar, or the mesh config?). Sidecar injection interacts with jobs, init containers, and startup ordering in ways that generate a steady trickle of platform tickets.

When I refuse:

• <20-30 services, or a mostly-monolith — NetworkPolicies + ALB/ingress + a retry library covers 80% at 5% of the cost
• No platform team capacity to own it — an unowned mesh is worse than no mesh
• The actual requirement is one feature (say mTLS only) — consider narrower tools (Linkerd for exactly this is defensible; Cilium transparent encryption; or SPIRE) before the full Istio surface

Adoption pattern when yes: Linkerd or Istio ambient for lower operational surface; onboard namespace-by-namespace with mTLS in permissive mode first; success metrics defined up front (mTLS coverage, MTTR delta, ticket rate).

One-liner: 'a mesh moves cross-cutting network concerns from N app teams to one platform team — adopt it when that trade is favorable, and not before.'

Core model: Git is the desired state; ArgoCD continuously compares live cluster state against it (reconciliation every ~3 min + webhooks) and syncs or reports drift. Rollback = revert commit. Audit = git log + Argo's own history.

Repo and app architecture:

• Separate app-source repos from deployment-config repos — CI builds an image and opens a PR bumping the tag in the config repo; humans (or automation) merge = deploy. Never let CI push directly to main of the config repo for prod
• Config repo layout: apps/<service>/base + overlays/{dev,staging,prod} (Kustomize) or per-env values files (Helm); ApplicationSets generate Argo Applications across envs/clusters from that structure — 30 services × 4 envs without 120 hand-written Application specs
• Hub-spoke for fleets: one ArgoCD (HA, its own cluster) managing remote clusters, or ArgoCD-per-cluster for isolation — hub is operationally simpler until cluster count or network topology says otherwise

Drift policy (nuance interviewers probe): selfHeal: true + prune: true for prod (manual kubectl changes revert in minutes — which is the point: the emergency-change path must go through git too, or be a break-glass with an alarm). Tolerate specific drift via ignoreDifferences for fields controllers own (HPA-managed replicas, injected sidecars, cert rotations) — otherwise Argo and controllers fight forever.

Secrets (GitOps's awkward corner): never plaintext in git. Options: External Secrets Operator (git holds only references; ESO pulls from Vault/ASM at runtime — my default), Sealed Secrets (encrypted-in-git, key custody burden), or SOPS+KMS. The pattern: git declares which secret, a runtime system delivers the value.

Promotion: environments promote by PR — the diff between staging and prod overlays IS the review artifact. Gate prod merges on staging soak (automated checks posting to the PR); progressive delivery inside the cluster via Argo Rollouts (canary analysis against Prometheus, auto-abort on SLO breach). Image-tag automation (Argo Image Updater or CI bots) keeps humans out of the mechanical bumps while keeping the merge as the control point.

Failure modes to name: sync waves/hooks for ordering (CRDs before CRs, migrations before app), app-of-apps bootstrap for disaster recovery (rebuild a cluster from one root Application), and monitoring Argo itself — a stuck sync controller means your deploy pipeline is down even though everything looks green.

Closing line: 'the win isn't automation, it's that the cluster's entire configuration is a reviewable, revertible, auditable artifact — drift stops being a mystery because the diff is always computable.'

The automatic timeline (know the defaults):

• kubelet heartbeats via Lease objects every ~10s; miss for 40s → node controller marks NotReady
• NotReady triggers taints (node.kubernetes.io/not-ready:NoExecute); pods have a default toleration of 300s — so pods are evicted/rescheduled ~5 minutes after the node goes dark (tunable per-pod via tolerationSeconds)
• Crucially: 'evicted' here means the API objects are deleted and replacements scheduled elsewhere; if the node is network-partitioned but alive, the pods may still be running — this is why StatefulSets won't force-replace pods on a NotReady node without fencing (split-brain protection)
• Endpoints: pods on the dead node drop from Service endpoints once marked not-ready — traffic stops routing to them well before eviction

Runbook:

1. Scope first: one node or many? Many = look up (AZ impairment, control-plane issue, cert expiry, CNI/daemonset rollout gone wrong) — don't debug one tree in a burning forest
2. Single node: cloud console/status checks (instance dead? network?), kubectl describe node conditions (MemoryPressure? DiskPressure? PLEG?), node logs via SSM/serial if reachable — common culprits: kubelet OOM (undersized system-reserved), disk full (image/log bloat), runtime hang
3. Bias to replace, not repair, during peak: cordon, confirm workloads rescheduled healthy, terminate the node (let ASG/Karpenter replace), keep it (or its logs/snapshot) for post-incident forensics if the cause is unknown
4. Verify stateful workloads specifically — anything with volume affinity to that node/AZ needs attention; force-delete of stuck pods on partitioned nodes only with fencing certainty (instance verified terminated)

Architecture so this pages nobody:

• Capacity: N+1 per AZ, topology spread constraints so no service concentrates on one node; PDBs so voluntary ops never collide with involuntary failures
• Tuning: for latency-sensitive fleets, lower tolerationSeconds (e.g. 60s) for faster failover — accepting more rescheduling churn on network blips
• Auto-remediation: node-problem-detector + a remediation controller (or Karpenter node health) auto-recycling NotReady nodes after a threshold; alert only on rate of node failures, not each one
• Stateful tier: replicated storage or managed services so 'a node died' never equals 'data unavailable'

What they're testing: the 40s/5min timeline, the partitioned-node subtlety (why stateful is special, why fencing exists), and whether your default posture is cattle (replace + investigate later) rather than heroic node surgery mid-incident.

Cluster Autoscaler (CA): works on pre-defined node groups (ASGs) — pending pods trigger scaling an existing group; it simulates scheduling against group templates. Mature and everywhere, but: fixed instance shapes per group (bin-packing limited by your group design), group-by-group scale-up retries are slow (minutes when the first choice lacks capacity), and scale-down is conservative.

Karpenter: watches pending pods and provisions individual right-sized nodes directly (EC2 Fleet API, no ASGs) from a NodePool spec listing allowed instance families/sizes/capacity types. Wins: seconds-faster provisioning, flexible instance selection (picks from hundreds of types → dramatically better spot availability and bin-packing), consolidation (actively replaces underutilized/expensive nodes with cheaper ones), and native spot interruption handling. It's the default answer on AWS today; CA remains right for other clouds with weaker equivalents or orgs wedded to ASG-based controls.

Provisioning design for spiky + cost-sensitive:

1. Tiered NodePools:
   • critical — on-demand only, broad instance families, taint so only tier-1 services (with matching toleration + priorityClass) land there; sized-by-limits so consolidation is gentle
   • general — spot-first with on-demand fallback (Karpenter weights), wide instance diversity (the #1 spot-reliability lever — 30+ instance types across families/sizes/AZs)
   • batch — pure spot, aggressive consolidation, low priorityClass so batch preempts first under pressure
2. Spot hygiene: interruption queue handling enabled (2-min warning → cordon+drain), PDBs and graceful shutdown in every service, checkpointing for long batch jobs; measure interruption rate per instance type and prune bad actors from the pool
3. Headroom for spikes: priority-negative placeholder pods (pause pods) as warm capacity — evicted instantly when real pods need the room; cheaper than static overprovisioning because Karpenter consolidates it away off-peak
4. Guardrails: NodePool limits (max CPU/mem) as blast-radius caps, do-not-disrupt annotations on interruption-intolerant pods, consolidation windows if churn during business hours causes noise, and drift management (Karpenter replaces nodes on AMI updates — patching becomes continuous)

Metrics that prove it works: pending-pod-age p95 (provisioning speed), spot interruption rate, cost per pod-hour trend, consolidation savings, utilization (requests/allocatable) per pool.

Senior close: 'CA scales groups you designed; Karpenter designs the node for the pod. The strategy above typically lands 50-70% of compute on spot with tier-1 fully protected — the discipline is in the taints, priorities, and PDBs, not the autoscaler choice.'

The abstraction stack: PVC = a pod's claim ('20Gi, RWO, class fast'); PV = the actual volume (statically pre-created or dynamically provisioned); StorageClass = the provisioning template (which CSI driver, parameters like gp3/io2, reclaimPolicy, volumeBindingMode); CSI driver = the plugin actually creating/attaching/mounting cloud volumes (EBS, EFS, etc. — in-tree drivers are gone; CSI is the only path).

Details that matter in production:

• volumeBindingMode: WaitForFirstConsumer — delays volume creation until the pod schedules, so the volume lands in the pod's AZ. Without it (Immediate mode), the volume picks an AZ first and pins all future scheduling of that pod to that zone — the root cause of half of all 'volume node affinity conflict' incidents
• Access modes are about nodes, not pods: RWO = one node (EBS); RWX = many nodes (EFS/FSx); RWOP = one pod. EBS being RWO+zonal drives the whole HA design below
• reclaimPolicy: Delete (PVC gone → volume gone) vs Retain (volume survives for manual recovery) — prod databases get Retain
• Volume expansion online (allowVolumeExpansion), CSI snapshots (VolumeSnapshot API) for backup workflows, and per-pod volumes via volumeClaimTemplates in StatefulSets

Design: stateful workload surviving AZ loss (say, Postgres):

The key realization: EBS doesn't cross AZs — so AZ survival must come from application-level replication, not storage.

1. Topology: 3 replicas via an operator (CloudNativePG/Patroni-based), one per AZ (topologySpreadConstraints + WaitForFirstConsumer), each with its own zonal EBS PV (gp3/io2 sized for IOPS)
2. Replication + failover: synchronous replication to at least one standby (RPO=0 for AZ loss), operator-managed leader election and promotion (RTO seconds-minutes); the service endpoint follows the primary
3. AZ-loss behavior rehearsed: primary's AZ dies → standby promotes; the old PV stays pinned to the dead AZ — the replacement replica gets a new volume in a healthy AZ and re-syncs from the new primary (capacity/time for re-sync is part of the design math)
4. Backups are separate from HA: scheduled base backups + WAL archiving to S3 (cross-region for DR), restore drills scheduled — snapshots of a corrupt database replicate the corruption; PITR is the real safety net
5. Guardrails: PDB (maxUnavailable 1), Retain reclaim policy, priorityClass, and monitoring on replication lag as the leading indicator

Alternative to state honestly: RWX/regional storage (EFS) sidesteps zonal pinning for some workloads, but databases on NFS-semantics storage is usually the wrong trade (latency, locking) — and 'use RDS/managed' is the right answer for most orgs unless platform depth or requirements say otherwise.

One-liner: 'in Kubernetes, storage HA is an application-architecture problem — the platform gives you zonal bricks; replication across them is on you (or your operator).'

Situation: API p99 went from 50ms to 3s; kubectl feels drunk; ArgoCD syncs and HPA decisions lag minutes behind. Managed control plane (EKS), so no 'add API servers' escape hatch — the demand side must be fixed.

Diagnosis path:

1. API server metrics: apiserver_request_total by verb/resource/client — the smoking gun is usually one client user-agent doing LIST storms. LISTs (especially unpaginated, label-selector-free, cluster-wide) are the most expensive verb: each one hits etcd (or the watch cache) and serializes megabytes
2. Audit logs (or APF debug): who exactly — which ServiceAccount, from which pods
3. Priority & Fairness (APF): check apiserver_flowcontrol_* — which flow schemas are queued/rejected; APF is why one abuser degrades everyone less than it used to, but a big enough storm still hurts

The usual suspects (know this bestiary):

• A custom controller/operator without informers — raw LIST-every-10s reconcile loops; the #1 offender. Fix: client-go informers/listers (one WATCH + local cache instead of thousands of LISTs)
• CI/CD or scripts doing kubectl get ... -A in tight loops across hundreds of pipelines
• A misconfigured agent (monitoring/security DaemonSet) each instance LISTing cluster-wide — N nodes × per-node LISTs = quadratic pain
• Controllers with a hot resync (resyncPeriod seconds instead of minutes/hours) or crash-looping operators re-LISTing on every restart
• Object bloat amplifying everything: giant ConfigMaps, thousands of stale objects, unbounded Events

Fixes, immediate → structural:

1. Contain: APF FlowSchema putting the offending SA into a low-priority, low-concurrency bucket (or in emergencies, RBAC-revoke / scale the offender to zero) — restores cluster health while you fix properly
2. Fix the client: informers with shared caches, pagination (limit/continue) on unavoidable LISTs, field/label selectors server-side, resourceVersion=0-aware reads where staleness is fine, sane resync periods, exponential backoff on errors (default client-go rate limits are per-client — qps/burst tuned down for background tools)
3. Systemic guardrails: per-team client budgets via APF flow schemas, controller code-review checklist (informer usage is a merge requirement), object-count/size hygiene (Event TTL, ConfigMap size lint), and API-server latency + top-client dashboards so the next abuser is a graph, not an outage

Senior signal: you name informers-vs-LIST as the core pattern, know APF is the isolation mechanism, and treat API capacity as a shared resource with budgets — 'the API server is a multi-tenant database; clients that table-scan it need fixing, and the platform needs quotas so one bad client can't take the floor out.'

Threat model first: you're defending against (a) compromised/typosquatted base images, (b) build-pipeline tampering, (c) registry compromise or tag mutation, (d) known-CVE images reaching prod, (e) drift at runtime (something exec'ing new binaries into a running container).

Build side:

1. Minimal, pinned bases — distroless/chainguard-style images, digests not tags, rebuilt on a cadence (a 'no rebuilds' policy means you're accumulating CVEs even with no code changes)
2. SBOM generation (syft) at build; CVE scanning (grype/trivy) with severity gates in CI — plus continuous re-scanning of deployed images against new CVE feeds (the scanner you ran at build time doesn't know about next month's CVE)
3. Sign artifacts: cosign (keyless with OIDC workload identity ties the signature to which pipeline built it); attach SBOM + provenance attestations (SLSA-style) so 'what is this image and who built it from what commit' is cryptographically answerable

Admission side (the cluster's border control):

• Policy engine (Kyverno / Gatekeeper + sigstore policy-controller) enforcing: images only from approved registries; signature verification required (per-namespace signer identity — prod requires the prod pipeline's identity); digest pinning (mutate tags→digests); no latest; vulnerability-attestation freshness for sensitive namespaces
• Enforce in audit mode first, then block — and design the emergency path (break-glass namespace with alarms) before you need it, or the first incident hotfix will be blocked by your own policy at 3am

Runtime side:

• readOnlyRootFilesystem + non-root (from the pod-security baseline) makes payload-drop persistence hard
• Falco/eBPF runtime detection for drift signatures: exec into containers, new binaries executing, unexpected outbound connections, crypto-miner patterns
• Image immutability discipline: kubectl exec in prod is an audited exception, not a workflow — anything 'fixed live' is drift by definition and gets redeployed properly

Registry hygiene: private registry with immutable tags (or digest-only), pull-through cache for upstream images (rate limits + availability + a scan/quarantine point), retention policies so 'what's actually deployed' stays auditable.

The maturity claim that lands: 'every image in prod is traceable to a signed build from a reviewed commit, admission rejects anything else, and runtime watches for the gap between what was admitted and what's executing. Supply chain security is a chain — build, admit, run — and it's only as strong as the weakest link you actually enforce.'

Why it's genuinely hard: the cloud bill says 'EC2: $180K' but pods share nodes, requests ≠ usage, and shared services (ingress, monitoring, control plane) belong to everyone and no one. Attribution is a modeling exercise, not a query.

The model:

1. Unit of attribution: namespace (mapped to team/service via mandatory labels — enforced by admission policy: no team + cost-center labels, no deploy)
2. Cost basis: max(requests, usage) per pod — charging pure usage rewards over-requesting (you reserved capacity others couldn't use); charging pure requests ignores burst reality. Max-of-both makes over-requesting and under-requesting visible
3. Node costs → pods: each node's hourly cost (with its real pricing — spot vs on-demand vs SP-covered) divided across resident pods by their share; idle/unallocated node capacity attributed to the platform as a line item — platform's incentive to consolidate
4. Shared services: monitoring, ingress, mesh, CI runners metered and allocated proportionally (by usage or headcount-weighted) — visible as 'platform tax' on every team's report, which forces the platform team to justify its overhead
5. Tooling: OpenCost/Kubecost (or cloud-native like CUDOS+CUR joins) feeding dashboards; reconcile monthly against the actual AWS bill — attribution that doesn't sum to the invoice loses trust instantly

Making it drive behavior (the actual point):

• Showback first, chargeback later — a quarter of visible-but-not-billed reports lets teams fix embarrassments before money moves; premature chargeback creates gaming and org warfare
• Each team's report: total, trend, unit cost (per request / per tenant / per build — negotiated per team), requested-vs-used waste, and top-3 actionable items (rightsize X, spot for Y, delete Z)
• Budgets + anomaly alerts per namespace; efficiency targets (utilization, waste %) in platform OKRs

Result pattern from doing this for real: the first report typically finds 20-30% quick wins (abandoned namespaces, 10x over-requested dev environments, forgotten load-test clusters) — visibility precedes optimization.

Senior close: 'the technical model matters less than two properties: it reconciles to the real bill, and every number on a team's report maps to an action they can take. Cost data nobody can act on is just sad accounting.'

The primitives and their direction of control:

• Taints (node repels) — a node says 'nothing schedules here unless it tolerates me'. Effects: NoSchedule, PreferNoSchedule, NoExecute (also evicts running pods). Taints protect nodes from pods.
• Tolerations (pod may enter) — permission, not attraction. A toleration alone doesn't place a pod on the tainted node.
• Node affinity (pod seeks nodes) — required (hard) vs preferred (soft) node selection by labels.
• Pod affinity/anti-affinity (pods seek/avoid pods) — 'spread my replicas apart' / 'co-locate with the cache'. Expensive at large scale (scheduler computes pairwise); prefer topology spread for spreading.
• Topology spread constraints — even distribution across zones/nodes with maxSkew, and crucially whenUnsatisfiable: DoNotSchedule|ScheduleAnyway.

The design for the mixed cluster:

1. GPU pool: taint nvidia.com/gpu=true:NoSchedule; GPU workloads carry toleration + node affinity + resources.limits: nvidia.com/gpu: 1. Taint+affinity together — taint keeps others out, affinity pulls GPU pods in; either alone is half a solution.
2. Spot pool: taint capacity=spot:NoSchedule; batch/stateless tolerate it; interruption-sensitive workloads simply lack the toleration — safe by default.
3. Compliance-pinned (e.g. PCI): dedicated nodes tainted + labeled; PCI pods require-affinity to them; policy engine (Kyverno) enforces that non-PCI namespaces can't add the toleration — tolerations are permissions and RBAC doesn't cover them.
4. Every tier-1 service: topology spread across zones (maxSkew 1, DoNotSchedule) + across nodes (ScheduleAnyway) — hard zonal balance, soft node balance.

Gotchas that show experience: affinity is evaluated at scheduling time only (no rebalancing later — use descheduler if drift matters); required-affinity + small node pools = self-inflicted Pending storms; NoExecute taints with tolerationSeconds are how node-pressure evictions are tuned.

One-liner: 'taints are fences, affinity is a magnet, spread is a leveler — real placement policy uses all three deliberately.'

The termination sequence (the part everyone half-knows):

1. Pod marked Terminating; two things start in parallel: (a) endpoint removal from Services propagates to every node's kube-proxy and every ingress/LB, (b) kubelet runs preStop hook, then sends SIGTERM to containers
2. After terminationGracePeriodSeconds (default 30s), SIGKILL

Why deploys drop requests: the race in step 1 — SIGTERM often arrives before every LB/kube-proxy has stopped sending traffic (propagation takes hundreds of ms to seconds across nodes, longer for cloud LB target deregistration). An app that exits immediately on SIGTERM closes connections that were legitimately routed to it a moment later → 502s.

The correct recipe:

• preStop sleep (5-15s): crude, universal, effective — the pod keeps serving while endpoint removal propagates, then gets SIGTERM. This one line fixes most deploy-time 5xx
• App SIGTERM behavior: stop accepting new connections, finish in-flight requests (with a deadline), drain keep-alives (send Connection: close), then exit. Fail readiness immediately on SIGTERM so any lagging balancer drops you
• Grace period sized to reality: p99 request duration + drain time + margin; long-polling/websocket services need 60s+, and matching ALB deregistration delay
• Long-lived connections: gRPC/websocket clients need GOAWAY/reconnect signaling — server-side draining alone can't fix a client that never re-resolves
• Jobs/consumers: SIGTERM → stop taking new work, checkpoint/ack current item; visibility-timeout math for queue consumers so a killed worker's message reappears

Verify it's real: deploy during load test; the acceptance criterion is zero 5xx during rollout, not 'usually fine'.

Senior signal: naming the endpoint-propagation race as the root cause — most teams cargo-cult the sleep without knowing why it works.

CrashLoopBackOff means: container starts, exits (non-zero or killed), kubelet restarts with exponential backoff (10s→20s→...→5min cap). The state is a symptom; the exit is the event.

The path:

1. kubectl describe pod — exit code + reason first:
   • Exit 137 = SIGKILL: OOMKilled (check lastState reason) or failed liveness probe killing it or grace-period expiry
   • Exit 1/2/custom = app error; 126/127 = command not found/not executable (bad entrypoint, missing binary, arch mismatch — arm64 image on amd64 node)
   • Also read Events: image pulled OK? volume mounts failing? config missing?
2. kubectl logs --previous — the previous crashed container, not the current empty one. This flag is half the battle
3. Logs empty? The process died before logging: entrypoint/command wrong, missing env/config at startup (kubectl get pod -o yaml — compare env/volumeMounts against what the app expects), missing ConfigMap/Secret key (describe shows CreateContainerConfigError variants), or instant segfault. Overridden entrypoint debugging: kubectl debug or patch command to ["sleep", "3600"], exec in, run the real entrypoint by hand and watch it fail interactively
4. Liveness-probe kills masquerading as crashes: describe shows probe failures; the app was fine but slow to start → add/lengthen startup probe
5. Init containers: kubectl logs pod -c <init-container> — the pod object hides which init failed unless you look
6. Environmental diffs: works in dev, crashes in prod → resource limits (JVM heap vs limit), missing IAM permissions on startup (IRSA misconfigured → SDK exception at boot), readonly rootfs the app didn't expect, or seccomp/PSA blocking a syscall

Backoff nuisance: during debug, the 5-minute backoff wastes time — kubectl rollout restart or delete the pod to reset it once you've changed something.

What they're testing: --previous, exit-code literacy, the sleep-entrypoint trick, and not blaming Kubernetes for what's a process that exits.

First, sort the failure modes — they need different tools:

1. Deleted/corrupted API objects (fat-fingered kubectl delete ns, bad operator) → GitOps repo restores declared state instantly; Velero catches what GitOps doesn't hold: PVC data, resources created by controllers at runtime, CRD instances from user actions
2. Persistent data loss (volume corruption, ransomware) → CSI snapshots via Velero + application-native backups (DB dumps/WAL to S3) — snapshots of a corrupted database are corrupted backups; app-level backup with PITR is the real line of defense for databases
3. Whole-cluster loss (region issue, cluster config disaster, botched upgrade) → rebuild-from-code strategy

Velero specifics worth knowing: backs up API objects (filtered by namespace/label) to object storage + orchestrates volume snapshots (CSI) or file-level copies (Kopia/restic — slower, but portable across storage classes/clouds); schedules + TTLs; restore can remap namespaces and storage classes (that portability is what makes cross-cluster restore work). Its gaps: doesn't quiesce applications (crash-consistent, not app-consistent, unless you add pre/post hooks running fsfreeze/DB checkpoints), and restore ordering of CRDs/webhooks can need care.

Cluster-loss runbook (the answer they actually want):

1. Provision: cluster from IaC (Terraform/eksctl) — 20-40 min; this is why cluster config lives in code, never hand-applied
2. Bootstrap: install ArgoCD via bootstrap script → app-of-apps root Application → platform add-ons then workloads reconcile from git in dependency order (sync waves)
3. Data: Velero restore for PVC-backed workloads into the new cluster; databases restore from app-native backups (measure this — it dominates RTO)
4. Cutover: DNS/global-accelerator repoint, certificates/secrets from external stores (ESO re-syncs — another reason secrets don't live in the cluster)

The discipline that separates real DR from paper DR: quarterly game-day executing this end-to-end with a measured RTO (target e.g. <2h) — teams that haven't rehearsed discover their backup IAM role was deleted, CRD ordering deadlocks Argo, or the DB restore takes 6 hours, live, during the real incident.

One-liner: 'GitOps restores what you declared, Velero restores what accumulated, app backups restore what matters most — and a DR plan is only as real as its last rehearsal.'

Why CPU-based HPA fails queue consumers: a worker processing SQS messages at steady 40% CPU with a backlog of 2 million messages looks healthy to HPA — CPU measures effort, not demand. The correct scaling signal is queue depth (or better, backlog ÷ processing rate = time-to-drain) — a leading indicator, where CPU is trailing.

KEDA's architecture: a metrics adapter + controller that turns external event sources into scaling signals — 60+ scalers (SQS, Kafka consumer lag, Prometheus queries, cron, Redis streams, cloud queues). Under the hood it creates and manages an HPA for you (ScaledObject → HPA with external metrics), so it composes with everything HPA does. Two things HPA alone can't do:

1. Scale to zero — below the activation threshold KEDA deletes all replicas and holds a watch on the source; first message wakes the deployment (cold-start latency = pod startup, so size images/startup accordingly)
2. Scale on things Kubernetes can't see — Kafka consumer group lag, cloud queue length, a PromQL expression over business metrics

Design for an SQS worker fleet:

• ScaledObject: queueLength target ~ messages-per-pod-per-poll-interval (e.g. target 5 in-flight per replica), min 0, max sized to downstream capacity (DB connections! — scaling consumers 10x turns a backlog into a database incident; the max replica bound is a bulkhead)
• Visibility timeout > p99 processing time; graceful shutdown acks/returns in-flight messages (pairs with the SIGTERM discipline)
• ScaledJobs instead of Deployments for long-running/run-to-completion work items (one job per message pattern) — avoids the 'scale-in kills a 2-hour job' problem
• Flap control: stabilization windows and cooldown so bursty queues don't oscillate replicas; cron scaler layered for known daily peaks (pre-warm before the 9am burst)

When plain HPA is still right: request/response services scale fine on RPS or CPU; KEDA earns its keep where demand lives outside the pod — queues, streams, schedules.

Senior close: 'scale on the signal closest to user-visible backlog, bound it by what downstream can absorb, and treat scale-to-zero as a latency trade you either accept or pre-warm around.'

Why it's harder than stateless: Kafka cares about identity (broker IDs), locality (data gravity — moving a broker means re-replicating terabytes), ordering of operations (rolling restarts must respect ISR health, not just pod readiness), and network stability (client connections address specific brokers, not a load-balanced pool). Kubernetes's default assumption — pods are interchangeable and disposable — is precisely false here.

The production setup:

1. Operator, non-negotiable: Strimzi (the de-facto standard) — because day-2 Kafka ops (rolling restarts that wait for ISR, rebalancing via Cruise Control, cert rotation, version upgrades in the correct broker/controller order) must be encoded, not hand-run. Raw StatefulSets for Kafka is a trap: StatefulSet rolling semantics know nothing about under-replicated partitions
2. Topology: brokers spread one-per-AZ-minimum (topology spread + rack.awareness mapped to zones so replicas never co-locate in an AZ); dedicated node pool (taints) — Kafka's page-cache hunger and IO patterns make it a terrible neighbor
3. Storage: local NVMe (best latency, but broker loss = full re-replication) vs EBS gp3/io2 via CSI (my default: survives pod rescheduling within the AZ — the volume re-attaches, no data re-sync) with WaitForFirstConsumer; throughput math done against both disk and network (replication traffic ≈ ingress × replication factor)
4. Networking: clients need per-broker addressability — headless service for internal; for external access, per-broker NodePort/LB listeners that the operator manages; never a plain LoadBalancer round-robining brokers
5. Disruption discipline: PDB maxUnavailable=1; do-not-disrupt/consolidation exclusions so Karpenter doesn't 'optimize' a broker away mid-day; graceful shutdown periods long enough for controlled leadership handoff; K8s upgrades coordinate with Cruise Control rebalancing
6. Capacity + monitoring specifics: under-replicated partitions (the #1 alert), ISR shrink rate, consumer lag, disk fill projections (log retention vs disk is a date-math alarm, not a percentage one), page-cache-aware memory sizing (heap small, cache large)

When to say no: small team, moderate throughput, no existing Kafka expertise → MSK/Confluent Cloud and spend the platform effort elsewhere. Run-it-yourself-on-K8s is justified by scale economics, data-locality requirements, or an org that already operates Kafka well on VMs and wants consolidation.

What they're testing: do you know which Kubernetes conveniences are actively dangerous for quorum systems (naive rolling updates, consolidation, LB-everything), and does your answer center the operator + rack awareness + PDB discipline rather than YAML trivia.