Docker

A VM virtualizes hardware — each VM runs a full OS with its own kernel on a hypervisor. A container virtualizes the operating system — processes share the host kernel, isolated by kernel features (namespaces + cgroups).

The consequences:

• Startup: VMs boot an OS (minutes); containers start a process (milliseconds)
• Density: container overhead is megabytes, not gigabytes — 10x+ more per host
• Isolation: VMs have a hardware-enforced boundary; containers share a kernel — a kernel exploit escapes all containers on the host. That's why hostile multi-tenancy uses VMs (or micro-VMs like Firecracker)

One-liner: 'a container is a well-dressed process; a VM is a whole computer. Choose isolation strength vs density/speed accordingly.'

• Image — an immutable, layered filesystem snapshot + metadata (entrypoint, env, ports): the template. Built once, content-addressed (digest), shareable via registries
• Container — a running instance of an image: the image's read-only layers plus one writable layer on top, plus a namespaced process

The class/object analogy holds: one image, many containers, each with its own writable layer, network identity, and lifecycle.

The details that show depth:

• The writable layer is ephemeral — docker rm and its contents are gone; anything worth keeping goes in a volume. This is the mechanical reason 'containers are stateless' is the default design stance
• Writes to files from image layers trigger copy-on-write — the file is copied up to the writable layer first (slow for large files, why databases write to volumes)
• docker commit can snapshot a container into an image — and is an anti-pattern for anything real (unreproducible 'pet images'); images come from Dockerfiles in CI, full stop
• Image identity: tags are mutable pointers (:v1.2 can be repushed); digests (@sha256:...) are immutable truth — production pins digests

FROM node:20-slim            # base image — the starting layer stack
WORKDIR /app                 # cd (creates dir) for following instructions
COPY package*.json ./        # copy files: dependency manifests FIRST (caching!)
RUN npm ci --omit=dev        # execute at BUILD time → new layer
COPY . .                     # app source after deps (changes most often)
ENV NODE_ENV=production      # runtime environment variable
EXPOSE 3000                  # documentation of the listening port (no networking effect)
USER node                    # drop root for runtime
ENTRYPOINT ["node", "server.js"]   # the command containers run

The distinctions interviewers probe:

• RUN vs CMD/ENTRYPOINT: RUN executes at build time creating layers; ENTRYPOINT/CMD define what executes at run time (nothing runs at build)
• COPY vs ADD: COPY copies; ADD also unpacks tar archives and fetches URLs — those surprises are why the rule is always COPY unless you specifically need tar extraction
• EXPOSE does nothing to networking — it's metadata; -p 8080:3000 does the actual publishing
• Ordering is a caching strategy, not style — see the layer-caching question
• USER: no USER instruction = runs as root — the most common Dockerfile security miss

Every layer-creating instruction (RUN, COPY, ADD) produces an immutable filesystem diff stacked on the previous ones. At build, Docker walks the Dockerfile top-down: if the instruction and its inputs are unchanged, the cached layer is reused — but the first cache miss invalidates every layer after it.

The rules that follow:

1. Order by change frequency: base → system deps → dependency manifests + install → source code. Code changes daily; deps weekly — the expensive npm ci/pip install layer should survive code-only rebuilds
2. COPY the manifest alone before installing — COPY package.json then RUN npm ci then COPY . .; copying everything first means any file change re-runs the install
3. Combine related RUNs (apt-get update && apt-get install -y ... && rm -rf /var/lib/apt/lists/*) — update and install in separate layers is a classic stale-cache bug, and cleanup in a later layer doesn't shrink the image (the files still exist in the earlier layer)
4. .dockerignore — a fat, churning build context (node_modules, .git) both slows uploads and causes spurious COPY invalidations

Both define what runs at container start; the interaction is the interview question:

• ENTRYPOINT — the fixed executable; docker run arguments are appended to it
• CMD — the default arguments (or default command if no ENTRYPOINT); replaced entirely by any docker run arguments

ENTRYPOINT ["python", "app.py"]
CMD ["--port", "8080"]

• docker run img → python app.py --port 8080
• docker run img --port 9000 → python app.py --port 9000 (CMD replaced)
• docker run --entrypoint bash img → override requires the explicit flag

The pattern: ENTRYPOINT = 'what this container is', CMD = 'reasonable default flags'. Utility images (ENTRYPOINT ["kubectl"]) behave like binaries; app images often use an entrypoint script (env prep, then exec "$@").

The trap — shell vs exec form: ENTRYPOINT python app.py (shell form) wraps in /bin/sh -c — your app becomes a child of sh, signals don't reach it (SIGTERM hits sh, which ignores it) → no graceful shutdown, 10s SIGKILL every stop. Always exec form (JSON array), and exec the final process in entrypoint scripts for the same reason.

Docker runs single containers per command; Compose declares a multi-container application in one file and manages it as a unit:

services:
  api:
    build: .
    ports: ["8080:3000"]
    environment:
      DATABASE_URL: postgres://db:5432/app
    depends_on:
      db: { condition: service_healthy }
  db:
    image: postgres:16
    volumes: ["pgdata:/var/lib/postgresql/data"]
    healthcheck:
      test: ["CMD-SHELL", "pg_isready"]
volumes:
  pgdata:

docker compose up builds, creates a shared network (services reach each other by service name — db resolves via built-in DNS), starts in dependency order, aggregates logs. down unwinds it.

What it's for: local development and CI — the whole stack in one command, versioned with the code. depends_on with condition: service_healthy (not bare depends_on, which only orders starting) is the flake-killer for tests.

What it's not: a production orchestrator — no multi-host scheduling, no self-healing rescheduling, no rolling deploys worth the name. The mental model: Compose is the dev/CI harness; Kubernetes/ECS is the production runtime — and keeping the Compose file faithful to production topology (same services, same env var names) keeps 'works on my machine' honest.

The writable container layer dies with the container — persistent data needs a mount:

• Named volumes (-v pgdata:/var/lib/postgresql/data) — Docker-managed storage (/var/lib/docker/volumes/...): lifecycle independent of containers, portable across container replacements, backed up via docker run --volumes-from patterns. The default for data (databases, uploads, caches)
• Bind mounts (-v $(pwd)/src:/app/src) — map a host path directly: what's on the host is what the container sees, live. The tool for development (hot-reload source mounting) and for host integrations (mounting /var/run/docker.sock, config files)
• tmpfs (--tmpfs /scratch) — RAM-backed, gone at stop: secrets that shouldn't touch disk, fast scratch space

The distinctions that matter in practice:

1. Volume at a path with existing image content → image content is copied into the empty volume (first mount only); bind mount → host dir shadows image content entirely (empty host dir = empty container dir — the 'where did my files go' bug)
2. Bind mounts couple you to host filesystem layout + permissions (UID mismatches between host user and container user are the eternal papercut); volumes abstract it
3. macOS/Windows performance: bind mounts cross a VM boundary — heavy I/O (node_modules!) in bind mounts is brutally slow; keep dependency dirs in volumes even in dev setups
4. docker volume prune vs the fact that anonymous volumes (from VOLUME instructions without names) accumulate silently — a disk-space archaeology classic

Default: the bridge network. Each container gets a network namespace with its own interface, veth-paired to a host bridge (docker0); containers get private IPs (172.17.x.x) and NAT out through the host.

-p 8080:80 = publish: host port 8080 forwards to container port 80 — implemented with iptables DNAT rules (plus a userland proxy for edge cases). External traffic hits host:8080 and lands in the container; without -p, the container is reachable only from the Docker host / same network.

The drivers:

• bridge — default; user-defined bridges (from docker network create / Compose) additionally give DNS by container name — the reason 'containers can't find each other by name' is usually 'they're on the default bridge' (which has no name resolution)
• host — no network namespace: the container is the host network (no -p needed, no isolation, port conflicts are yours) — for performance-critical or network-tooling containers
• none — no networking: batch jobs, security-sensitive processing
• overlay — multi-host virtual networks (Swarm-era; K8s uses CNI instead)

Interview follow-ups to be ready for: container-to-container on the same user-defined network needs no -p (that's only for external exposure); published ports bypass typical host firewall expectations (the iptables rules Docker inserts often surprise security reviews); and 127.0.0.1:8080:80 binds the publish to localhost only — the flag people forget on dev boxes with public IPs.

docker exec -it <container> sh starts an additional process inside the container's existing namespaces — same filesystem, network, and process view as the app. It's the primary live-inspection tool.

The debugging toolkit in order:

1. docker logs (-f, --tail 100) — stdout/stderr history; works on stopped containers too, which makes it the first stop for crashes
2. docker inspect — full config + state JSON: exit code, OOMKilled flag, mounts, env, IP. docker inspect --format '{{.State.ExitCode}} {{.State.OOMKilled}}' answers 'why did it die' fast (137 + OOMKilled=true tells the story)
3. docker exec — poke the live container: check files, hit localhost endpoints, inspect env
4. Crashed container: can't exec into a dead one — docker logs it, or docker commit the corpse and run a shell in the snapshot (docker run -it --entrypoint sh <snapshot>) to examine the filesystem post-mortem
5. Minimal images with no shell (distroless/scratch): exec has nothing to run — use docker debug (or K8s ephemeral containers) to attach a toolbox; or docker cp files out; this trade-off is by design (no shell for attackers either)
6. docker stats, docker events, docker diff — live resource usage, daemon event stream, and what files changed vs the image (drift/debug forensics)

The habit that marks production experience: exit code literacy — 137 = SIGKILL (OOM or stop-timeout), 139 = segfault, 126/127 = entrypoint problems — inspect-then-logs answers most 'it just died' tickets in two commands.

A registry stores images as content-addressed blobs (layers) + manifests (the JSON tying layers together per architecture). docker push/pull move only missing layers — that's why a one-line change to a well-ordered Dockerfile pushes in seconds (one new layer) and why base-image reuse across services saves bandwidth and disk everywhere.

Identity, precisely:

• registry.corp.io/payments/api:1.4.2 — registry / repository / tag. Tags are mutable pointers: whoever can push can re-point 1.4.2 at different content tomorrow
• ...@sha256:abc123 — digest: the content hash of the manifest; immutable by construction. Same digest = bit-identical image, forever
• :latest is just a tag convention — it means 'whatever was pushed without a tag', not 'newest', and it makes deployments unreproducible (which latest did you run last Tuesday?)

Production discipline that follows:

1. Build once, promote the digest — CI builds, tests, then the same digest moves dev → staging → prod (retag or record; never rebuild per environment — a rebuild is a different artifact no matter how similar)
2. Deployments pin digests (or immutable-by-policy tags); registries support tag-immutability settings — turn them on
3. Registry operations are production operations: retention policies (untagged/old digests accumulate terabytes), replication for DR, pull-through cache for Docker Hub (rate limits + availability), and auth via short-lived tokens, not shared passwords

One-liner: 'tags are for humans, digests are for machines — deploy by digest, keep tags as friendly labels, and treat the registry as tier-1 infrastructure.'

The mechanism: multiple FROM stages in one Dockerfile; later stages COPY --from=<stage> selected artifacts. Only the final stage ships — build toolchains, source, and intermediate artifacts stay behind.

FROM golang:1.22 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download            # cached unless mods change
COPY . .
RUN CGO_ENABLED=0 go build -ldflags='-s -w' -o /out/api ./cmd/api

FROM gcr.io/distroless/static-debian12 AS runtime
COPY --from=build /out/api /api
USER nonroot
ENTRYPOINT ["/api"]

The 1.2GB → 20MB math: golang:1.22 is 800MB of toolchain + your source + deps — none of which the running binary needs. A static binary on distroless/static (2MB base) ships the binary, CA certs, tzdata, nothing else.

The patterns beyond size:

• Test stage: FROM build AS test; RUN go test ./... — CI targets it (--target test); prod builds skip it. One Dockerfile, multiple pipeline uses
• Dev stage: a stage with hot-reload tooling for Compose to target — dev/prod stay in one file, guaranteed same base
• Parallelism: BuildKit builds independent stages concurrently; only stages the target needs are built at all
• COPY --from=<external-image> — lift artifacts from any image (a specific binary from an official image) without stage-building it

Interview extras that land: interpreted languages benefit too (builder installs dev deps + compiles assets; runtime copies site-packages/dist only), and the security framing — the deploy image contains no compiler, no shell, no package manager: dramatically less for an attacker to live off.

Situation: CI builds take 12 minutes (every build re-downloads dependencies), and an API token was found baked into an image layer via a build ARG.

The BuildKit toolkit (default engine now; docker buildx fully unlocks it):

1. Cache mounts — fix the re-download:

RUN --mount=type=cache,target=/root/.cache/pip pip install -r requirements.txt
RUN --mount=type=cache,target=/root/.m2 mvn package

The package-manager cache persists across builds without entering any layer — layer caching still governs whether the step re-runs, but when it runs, it's incremental. Typical: minutes → seconds for dependency steps 2. Secret mounts — fix the token leak:

RUN --mount=type=secret,id=npm_token \
    NPM_TOKEN=$(cat /run/secrets/npm_token) npm ci

docker build --secret id=npm_token,env=NPM_TOKEN — the secret is available during the RUN, in tmpfs, and never enters a layer. Build ARGs, by contrast, are baked into image history (docker history shows them) — the leak class you found. Same for --mount=type=ssh for private git 3. Remote/registry cache — fix cold CI runners:

docker buildx build --cache-from type=registry,ref=corp.io/api:cache \
                    --cache-to type=registry,ref=corp.io/api:cache,mode=max .

Layer cache exported to the registry — ephemeral runners import it instead of building cold. mode=max caches intermediate stages too (multi-stage needs it); GHA has a native cache backend (type=gha) 4. Supporting cast: --mount=type=bind for build-only file access without COPY layers; parallel stage execution (free speedup for well-factored multi-stage files); docker buildx bake for coordinated multi-image builds

Result shape from doing this: 12min → ~2min warm / ~5min cold, secret out of image history (rotate it — history is forever in already-pushed images), and the cache registry repo becomes a build asset you manage (retention!).

One-liner: 'BuildKit separates step caching from content caching, and moves secrets out of the layer model entirely — if your CI rebuilds dependencies or passes tokens as ARGs, you're using 2018 Docker in a 2026 pipeline.'

'Container' is not a kernel object — it's a process wearing kernel-enforced constraints. Docker/containerd assemble them:

Namespaces — what a process can SEE (isolation):

• pid — own process tree; the container's PID 1 is just a host process with a second identity
• net — own interfaces, routes, iptables (the veth-to-bridge story)
• mnt — own mount table; the image filesystem is a mount namespace showing the overlayfs stack
• uts (hostname), ipc (shared memory), user (UID remapping — container root ≠ host root when used; the underused one), cgroup (own cgroup view), time (rare)

cgroups — what a process can USE (resource control): hierarchical controllers metering and limiting CPU (shares/quotas — throttling), memory (limits — OOM kill), IO, pids (fork-bomb caps). docker run --memory=512m --cpus=2 writes cgroup values; the kernel enforces.

The rest of the sandwich:

• overlayfs — image layers union-mounted read-only + writable upper dir = the container filesystem (copy-up on write)
• capabilities — root's powers split (~40 flags); Docker drops most by default (keeps NET_BIND_SERVICE, drops SYS_ADMIN etc.)
• seccomp — syscall filter (default profile blocks ~44 of ~300+)
• LSM (AppArmor/SELinux) — mandatory access control on top

Why this matters beyond trivia: ps aux on the host shows container processes (they're just processes — this surprises people and is a forensics gift); a kernel bug pierces all containers at once (VM/micro-VM for hostile tenants); and 'containers are lightweight' is precisely because there's no guest OS — just bookkeeping on shared-kernel processes.

The interview flex: you can build a crude container in a shell — unshare --pid --net --mount --fork chroot ... + cgroup writes; Docker's value is the packaging ecosystem (images, registries, API), not kernel magic it invented.

Threat model: app compromise (RCE in your code/deps) → attacker lives in the container → escalation attempts: to root-in-container, to the host (escape), laterally via network/creds.

The hardening stack, orderd by leverage:

1. Non-root user (USER app, UID > 0): most escape techniques and file-permission abuses need in-container root. Build images with a dedicated UID; runtime enforcement (K8s runAsNonRoot: true) catches images that forgot
2. Drop capabilities: default Docker keeps ~11; production wants --cap-drop=ALL + add back the provable minimum (usually nothing; occasionally NET_BIND_SERVICE — or just listen on >1024). CAP_SYS_ADMIN is 'root by another name'; anything requesting it gets a design review
3. --read-only root filesystem + explicit tmpfs/volumes for writable paths: payload-drop and persistence get much harder; also documents the app's write surface
4. seccomp: keep the default profile (blocks the exotic syscalls real exploits use — unshare, kexec, bpf); --security-opt seccomp=unconfined in prod is a finding. Custom tighter profiles for high-value targets
5. no-new-privileges (--security-opt no-new-privileges) — blocks setuid escalation paths
6. Rootless mode / user namespaces: the daemon itself (or the container's root) maps to an unprivileged host UID — an escape lands as nobody, not root. Rootless Docker/Podman for CI runners especially (CI executes semi-trusted code)
7. Resource limits always (memory, pids) — a fork bomb or memory balloon is an availability attack; limits make it a contained failure

The socket rule: mounting /var/run/docker.sock into a container = handing it root on the host (it can start privileged containers). Whole categories of 'Docker-in-Docker for CI' designs fail here — prefer rootless builders (BuildKit rootless, kaniko-style) or isolated build VMs.

Verification, not vibes: docker inspect diffs against a golden security config in CI; runtime detection (Falco rules for exec-into-prod, unexpected outbound, privilege syscalls) because prevention without detection is half a posture.

One-liner: 'assume the app gets popped; the container config decides whether that's a bad Tuesday or a hosted-takeover — non-root, no caps, read-only, default seccomp, never the socket.'

The problem: in a container, your app is PID 1 — a role Unix normally gives to a real init system with two special duties your app doesn't perform:

1. Reaping zombies: when a process's parent dies, the orphan re-parents to PID 1; when it exits, PID 1 must wait() on it or it stays a zombie (dead but occupying a process-table slot). Apps that spawn subprocesses (shell-outs, chrome headless, ffmpeg workers) without reaping accumulate zombies → pid exhaustion → 'cannot fork' failures that look like memory issues
2. Signal defaults: normal processes die on unhandled SIGTERM; PID 1 ignores signals it hasn't explicitly handled. So docker stop sends SIGTERM → your unprepared PID 1 ignores it → 10s grace → SIGKILL. Every stop is a hard kill: no connection draining, no checkpoint, corrupted shutdown for anything stateful

The shell-form multiplier: CMD python app.py (shell form) makes sh -c PID 1 — sh doesn't forward signals to children, so even a signal-aware app never hears SIGTERM. Exec form + exec in entrypoint scripts ensures the app is actually PID 1 (or properly signaled).

The fixes:

• docker run --init (or init: true in Compose) — injects tini, a 4KB init that reaps zombies and forwards signals; the app runs as its child with normal signal semantics. The cheapest correct answer
• Or bake tini/dumb-init as ENTRYPOINT in the base image (platform teams: put it in the golden base)
• Or the app genuinely handles it: signal handlers + child reaping (some runtimes/frameworks do; most don't — verify, don't assume)
• Kubernetes note: same physics — pause container handles some namespace concerns but your container's PID 1 signal behavior is still yours; preStop hooks don't fix an app that ignores SIGTERM

How this surfaces in interviews and incidents: 'deploys drop requests' (no graceful shutdown → in-flight requests die with SIGKILL), 'container slowly breaks over days' (zombie accumulation), 'works locally, misbehaves under orchestration' (local runs short-lived; prod runs long enough to hit both).

One-liner: 'PID 1 is a job description, not just a number — either hire tini for it, or make sure your app actually performs the duties.'

The contenders:

• Full distro (debian/ubuntu): everything works; 100-400MB of attack surface and CVE-scanner noise you'll triage forever
• -slim: distro minus docs/extras (~80MB) — the pragmatic default for interpreted languages; apt still there for build stages
• Alpine (~5MB): tiny, has a shell/apk — but musl libc: glibc assumptions break subtly (DNS resolution differences historically, Python wheel availability — musl wheels exist now but lag; native modules recompile). Great for Go/static things; test carefully for Python/Node native-dep stacks
• Distroless (~2-20MB): runtime + your app, no shell, no package manager — the exploit-mitigation and CVE-noise win; debugging requires ephemeral-container tooling (a solved problem — see docker debug / kubectl debug)
• Chainguard/Wolfi-style: distroless philosophy + daily-rebuilt, near-zero-CVE images with SBOMs — where the industry is heading

The strategy that matters more than the pick — a golden-base program:

1. One blessed base per runtime (corp/base-python:3.12, corp/base-jre:21), built on the chosen upstream, adding: CA certs, tzdata, non-root user, tini, org labels. Teams FROM these, never raw upstream
2. Rebuild cadence is the security control: bases rebuild weekly + on CVE trigger; app images rebuild on base change (registry webhook → pipeline trigger). An image built once and never rebuilt accumulates CVEs at the base layer no code change will ever fix — 'no rebuilds' IS a vulnerability policy, just a bad one
3. Enforcement: admission/CI policy — only corp bases allowed (image provenance check), age limit on running images (force the rebuild treadmill), scanner gates tiered by severity + fix-availability
4. Digest-pinned upstreams in the base build (FROM debian@sha256:...) with renovate bumps — reproducibility and freshness, explicitly reconciled

The size honesty: size correlates with attack surface and pull latency, but the marginal MB matters less than people think — going 400→80MB matters; 80→20MB is mostly aesthetics unless cold-start pull time is a real constraint (serverless/spot churn). Optimize CVE surface and rebuild freshness first, bytes second.

One-liner: 'pick slim-or-distroless per runtime, but the real program is one golden base, rebuilt weekly, enforced at admission — freshness beats minimalism, and both beat artisanal FROM lines in 200 repos.'

The problem: CI jobs (often themselves containers) need to build images. The three roads:

1. Socket mount (-v /var/run/docker.sock): the job talks to the host's daemon. Fast, shared cache for free — and the job now has root on the runner host (can start privileged containers, read other jobs' containers/secrets). Acceptable only on single-tenant, trusted-pipeline runners; a finding on shared runners
2. Docker-in-Docker (dind service): a real daemon inside a privileged container per job. Isolated-ish from other jobs, but --privileged pokes holes to the host anyway, layer cache starts cold every job (unless cached via registry), and overlayfs-on-overlayfs quirks. The GitLab-era default, increasingly legacy
3. Daemonless/rootless builders (the modern answer): BuildKit rootless, kaniko (executes Dockerfile in userspace, no daemon, no privileges), buildah — image building as an unprivileged process. Composes with registry-based caching; runs on shared/multi-tenant runners without host-root risk

My default design:

• Builder: buildx targeting a rootless BuildKit (or kaniko where the platform dictates); no privileged containers anywhere in the standard path
• Cache: --cache-from/--cache-to type=registry,mode=max per repo (+ GHA/native cache where applicable) — ephemeral runners, warm builds; cache repo has retention policies (it grows unboundedly otherwise)
• Auth: OIDC-federated short-lived registry tokens (no long-lived robot passwords in CI secrets); push rights scoped per-repo
• The build produces, in one pass: image pushed by digest, SBOM (syft), vulnerability scan (fail on critical-with-fix), signature + provenance attestation (cosign keyless bound to the CI workflow identity). The digest + attestations are the pipeline's output contract — deploy stages consume the digest, admission verifies the signature
• Multi-arch via buildx QEMU or native arm64 runners (QEMU is 5-10x slower — native runners for hot paths, emulation for the long tail)
• Reproducibility hygiene: pinned base digests, SOURCE_DATE_EPOCH for stable timestamps, locked dependency manifests — same inputs, same image (or the diff is explainable)

The line that lands: 'image builds execute arbitrary code from PRs — treat the builder as a sandbox for hostile input, which rules out anything holding host-root: no socket mounts on shared runners, no privileged dind; rootless builders + registry cache give you speed without the blast radius.'

The setup: same image, dies in prod with exit 137 / OOMKilled — 'but it only uses 300MB on my machine.'

The usual suspects, in diagnostic order:

1. Limits exist in prod, not locally: docker run unlimited vs prod's --memory=512m (or K8s limits). First check: docker inspect --format '{{.HostConfig.Memory}}' / the pod spec. Local 'works' means 'nobody was counting'
2. The runtime can't see the limit (the big one): JVMs, and anything sizing itself from host memory — a JVM on a 64GB host defaults its heap from host RAM; inside a 512MB cgroup that's a death sentence. Modern runtimes are container-aware (JVM MaxRAMPercentage, .NET, Node --max-old-space-size still manual) — verify the flags, don't trust defaults, and remember: heap is not the whole process (metaspace, threads, direct buffers — heap should be ~60-70% of the limit)
3. cgroup accounting counts more than your app's RSS: page cache from file I/O (log-heavy apps), tmpfs mounts (emptyDir medium: Memory — files there are your memory), shm. The dashboard plots working set; the OOM killer reads the full cgroup — the gap is where 'flat graph, still OOMed' lives
4. Load shape differs: prod concurrency = N× per-request buffers; local single-user testing never allocates the peak. Load-test with prod limits locally (docker run --memory=512m + load) — reproduce before theorizing
5. Burst vs sample: metrics every 30s miss a 5s allocation spike; the kernel doesn't. dmesg/node OOM records show the cgroup's state at kill time — ground truth vs the graph's lie

The fix hierarchy: right-size the limit from measured P99 + headroom (not the local anecdote); configure the runtime to respect the cgroup explicitly; move page-cache-heavy workloads' expectations (limits account cache — either headroom for it or reduce file churn); and set requests=limits for memory in K8s so the number you tested is the number you get.

The reproduction habit that separates seniors: docker run --memory=<prod-limit> <image> + realistic load is a 5-minute local experiment that converts 'mysterious prod OOM' into a observable, debuggable event — most people theorize for hours instead of constraining locally for minutes.

One-liner: 'same image ≠ same process — memory behavior is image × limits × runtime-awareness × load; reproduce with prod's cgroup locally and the mystery usually solves itself.'

The mechanism: overlayfs union-mounts the image's read-only layers (lowerdirs, stacked) + one writable upperdir (the container layer) into a single merged view. Reads fall through the stack top-down (first hit wins); writes go to upper; deletes create whiteout files masking lower content.

The cliffs, and where they bite:

1. Copy-up on first write: modifying a file from a lower layer copies the entire file to upper first — append one log line to a 2GB file from the image, pay a 2GB copy, once, at an unpredictable moment (first write). Anything write-heavy (databases, caches) belongs on a volume (volumes bypass overlayfs entirely — direct host filesystem I/O); this is the mechanical reason, beyond persistence, that data goes in volumes
2. Layer-count depth: every read that misses upper walks lowerdirs; 60-layer images (bad Dockerfile hygiene, chained RUN sprawl) tax metadata operations. Multi-stage + combined RUNs keep depth sane (also: 127-layer hard limit exists)
3. Metadata-heavy workloads: npm install-style storms (100K small files, stat-heavy) in the container layer stress overlayfs inode handling — noticeably slower than a volume; CI that installs dependencies at runtime (rather than baked into the image or a cache mount) pays this constantly
4. docker diff / commit slowness on churned containers — enumerating upper vs lower at scale
5. Disk-space forensics: deleted-in-later-layer files still occupy earlier layers (image size ≠ sum of what's visible); /var/lib/docker fills from: dangling images, stopped containers' upper dirs, build cache, anonymous volumes, and logs (json-file driver without rotation — docker logs reading a 40GB file is its own incident). docker system df -v before prune

Practical rules that fall out: write-hot paths → volumes/tmpfs; dependency installs → image layers or BuildKit cache mounts, never runtime; log rotation configured at the daemon (max-size, max-file) not discovered at disk-full; and image hygiene (multi-stage, ordered layers) is runtime performance work, not just build aesthetics.

One-liner: 'overlayfs makes images cheap by making first-writes and deep stacks expensive — know which paths in your container are write-hot and route them around the union.'

Docker's native HEALTHCHECK:

HEALTHCHECK --interval=10s --timeout=3s --start-period=30s --retries=3 \
  CMD curl -fsS http://localhost:3000/health || exit 1

Runs inside the container; status (starting/healthy/unhealthy) surfaces in docker ps and events. Crucially: plain Docker does nothing on unhealthy — no restart, no traffic removal (restart policies key on exit, not health). Consumers that do act: Compose depends_on: condition: service_healthy (the real value — ordered, gated startup in dev/CI), Swarm (replaces unhealthy tasks), and various platform runners.

Kubernetes ignores HEALTHCHECK entirely — probes are pod-spec config (liveness/readiness/startup), executed by the kubelet. So a container relying on Dockerfile HEALTHCHECK for its health story is unmonitored on K8s until someone writes probes.

The design that works everywhere — put health in the APP, config in the platform:

1. Ship health endpoints as application contract: /health/live (process sanity only — no dependency checks; this maps to liveness) and /health/ready (can I serve: dependencies, warmup state — maps to readiness). Documented, versioned, cheap (<10ms, no auth)
2. Dockerfile HEALTHCHECK targets the liveness-ish endpoint with a generous start-period — this serves Compose/CI ordering and local dev signal
3. K8s manifests define probes against the same endpoints — same semantics, platform-appropriate tuning (probe thresholds, startup probe for slow boots)
4. The curl problem: distroless images have no curl/wget for the CMD — options: a tiny static healthcheck binary baked in, the app's own --healthcheck subcommand (argv self-check mode), or accept that K8s probes (http natively from kubelet — no in-container binary needed) are the real consumer and skip HEALTHCHECK for distroless
5. Semantics discipline transfers: liveness must not check dependencies (restart storms on downstream blips — same rule as K8s), readiness must reflect real serveability, and both must be tested (kill the DB in staging: readiness should fail, liveness should NOT)

One-liner: 'health is an application feature with two endpoints; HEALTHCHECK and probes are just two platforms consuming it — design the contract once, configure it per platform, and never let liveness look past the process boundary.'

The layering mechanism:

• compose.yaml — the canonical topology: all services, prod-faithful env var names, healthchecks, networks
• compose.override.yaml — auto-merged for local dev: bind-mount source for hot reload, expose debug ports, dev commands
• compose.ci.yaml — explicit (-f compose.yaml -f compose.ci.yaml) for CI: no mounts, built images, test runners
• Profiles — profiles: ["full"] on heavy/optional services: docker compose up starts the core; --profile full adds the kafka/search/worker tier. Solves 'the laptop can't run all 10' without forked files

The patterns that make 10 services livable:

1. Hybrid local/remote dev: developers run their service from source + its direct deps locally; the rest resolve to a shared dev environment (env-var-switched URLs). Full-stack-on-laptop stops scaling around service #6 — design the escape hatch deliberately rather than letting everyone invent one
2. depends_on with service_healthy everywhere — plus real healthchecks on infra services (postgres pg_isready, kafka topic check). 'Flaky local startup' is almost always missing health gating
3. Data lifecycle: named volumes for dev databases + a make reset that drops volumes and re-seeds (versioned seed scripts) — reproducible state beats 'my local DB is weird' archaeology
4. One .env file convention (gitignored, .env.example committed) for ports/credentials — onboarding is cp .env.example .env && docker compose up

Keeping parity honest (the senior concern):

• Same images where possible (dev target of the same multi-stage Dockerfile, not a different Dockerfile), same env var names as prod (values differ; names diverging = config bugs that only exist in one world), same network topology (service names = prod DNS names)
• Accept and document the gaps: no service mesh locally, single-instance everything, no IAM — a PARITY.md listing what dev doesn't replicate turns 'works locally, fails in staging' from mystery into checklist
• CI runs the compose stack for integration tests — the compose file itself is thereby tested, not drifting decoration

One-liner: 'Compose is the development harness: one canonical file, layered overrides, health-gated startup, and an explicit written line where local fidelity ends — parity you don't document is parity you don't have.'

The layered walk (each step isolates a failure class):

1. Name resolution first: docker exec A nslookup B (or getent hosts) —
   • Fails + both on default bridge → that's the bug: default bridge has no DNS; put them on a user-defined network (Compose does automatically)
   • Fails on a shared custom network → are they actually on the same network? docker network inspect <net> — Compose multi-file setups and docker run without --network create surprise topologies
2. Reachability: docker exec A ping <B's IP> (from inspect) — resolves-but-unreachable vs not-resolving are different bugs. No ping across networks = they're on different bridges (isolation by design — attach B to both networks or consolidate)
3. Port listening: docker exec B ss -tlnp — is the app listening, and on what interface? The classic: app bound to 127.0.0.1 inside the container — reachable from itself, invisible to the network. Containers must bind 0.0.0.0; this one line resolves a startling fraction of these tickets
4. App-level: docker exec A curl -v http://B:port/health — connection refused (nothing listening / wrong port), timeout (network path / firewall), HTTP error (app-level, network's fine). The distinction directs the next hour
5. Host-boundary cases (external → container): published port checks — docker port B, iptables DNAT present? Host firewall (firewalld/ufw vs Docker's iptables — their interaction is version-dependent and a known snake pit)? Bound to 127.0.0.1:8080:80 when you needed external access?
6. The exotic tier: MTU mismatches (VPN environments — large payloads hang, small requests work: the 'curl works, upload hangs' signature), conntrack exhaustion under load, inter-container iptables (com.docker.network.bridge.enable_icc=false), IPv6/IPv4 binding mismatches

Tooling for minimal images: no shell in distroless B — debug from a sidecar-style toolbox sharing the namespace: docker run --rm -it --network container:B nicolaka/netshoot — full tooling, B's exact network view, zero image changes. Know this trick; it converts 'can't debug minimal images' into a non-problem.

One-liner: 'walk DNS → IP → port → app in order, remember the two chronic offenders — default-bridge DNS and 127.0.0.1 binds — and keep netshoot in your pocket for shell-less images.'

The layer cake, top down:

• Docker (the CLI/daemon): developer UX — build, run, compose. Since 2017 it's a client of containerd underneath
• containerd: the container runtime daemon — pulls images, manages storage/snapshots, supervises container lifecycle via its API. What Kubernetes actually talks to (via CRI); what ECS/Fargate run; graduated CNCF, embedded everywhere
• runc: the low-level executor — one binary that takes an OCI bundle (rootfs + config.json) and makes the kernel calls (namespaces, cgroups, pivot_root) to start one container, then exits. Alternatives slot in here: gVisor (userspace kernel — syscall interception), Kata/Firecracker (micro-VM per container) for stronger isolation
• OCI specs — the reason this all interoperates: image-spec (what an image is: manifests, layers, config), runtime-spec (what runc executes), distribution-spec (registry API). An image built by anything (Docker, buildah, kaniko, bazel) runs on anything (containerd, CRI-O, podman) — the specs are the contract

Why 'Kubernetes removed Docker' confused everyone (and shouldn't have): K8s removed dockershim — the adapter that let kubelet talk to the Docker daemon — in favor of talking CRI directly to containerd. Images never changed (OCI images were always the artifact); docker build output runs exactly as before. The daemon was redundant middleware on nodes, not the image format

Where this knowledge pays operationally:

1. Node debugging changed: no docker CLI on modern K8s nodes — crictl ps, ctr, nerdctl are the node-side tools; runbooks written for docker ps on nodes are stale
2. Alternative runtimes are a K8s config (RuntimeClass): gVisor/Kata per-workload for multi-tenant or high-risk pods — you choose isolation strength per pod, and knowing the layer where that plugs in is the difference between using it and fearing it
3. Rootless/daemonless tooling (podman, buildah, BuildKit standalone) — all fruits of the OCI decomposition; CI security postures depend on them
4. Vendor-neutrality of your artifacts: betting on OCI images + registries is safe because of this structure — the build tool and runtime are swappable ends of a stable contract

One-liner: 'Docker is a UX, containerd is the engine, runc is the ignition, OCI is the standard that lets you swap any of them — and knowing which layer a problem lives in is increasingly what "knows containers" means.'

Situation: cost push to Graviton/ARM nodes (20-40% price-perf win), but 200 services ship amd64-only images — pods scheduled on ARM nodes crash with exec format error.

The mechanics — manifest lists: a multi-arch 'image' is a manifest list (OCI image index): one tag pointing at N per-architecture manifests; the runtime pulls the matching one automatically. Same tag works everywhere — the consumer never thinks about it.

docker buildx build --platform linux/amd64,linux/arm64 \
  --cache-from type=registry,ref=corp.io/api:cache \
  -t corp.io/api:1.4.2 --push .

Build strategies, by cost/speed:

1. QEMU emulation (buildx default via binfmt): zero infra, but 5-15x slower for compile-heavy builds — fine for the long tail, painful for hot paths
2. Native runner pool: arm64 CI runners (Graviton spot instances are cheap) building the arm64 half; buildx combines into one manifest list. The production answer for frequently-built services
3. Cross-compilation where toolchains allow (Go: GOARCH=arm64 in an amd64 builder + --platform=$BUILDPLATFORM staged Dockerfiles) — native speed, no extra runners; the xx helper images make this clean for C-dependent builds

The pitfalls that actually bite:

• Base images and dependencies: your base must itself be multi-arch (official ones are; that 2019 internal golden base isn't until you rebuild it); native npm/pip dependencies need arm64 wheels/prebuilds — audit before promising dates; the one C-extension without arm64 support becomes the whole migration's critical path
• Digest discipline changes: the manifest-list digest ≠ per-arch manifest digests — pinning tooling, vulnerability scanners, and signing must handle the index (cosign signs the index; scanners must scan both architectures — an arm64-only CVE is invisible if you scan amd64 only)
• Test both architectures: CI that tests amd64 and ships arm64 is shipping untested software — at minimum, smoke/integration suites on arm64 runners for tier-1 services; JIT-heavy runtimes (JVM) have arch-specific perf characteristics worth a load-test pass
• The rollout: ARM node pools tainted; services opt in via toleration after their multi-arch image + arm64 test pass lands — never flip a shared pool's architecture under running workloads

One-liner: 'multi-arch is a manifest list plus honest CI — the build mechanics are a flag; the real work is dependency audits, testing both halves, and scanning/signing the index, not just the arch you develop on.'

The problem set: hundreds of nodes pulling images (Hub rate limits: 100 anonymous/6h per IP — a NAT'd cluster exhausts that in one deploy wave), terabytes of accumulated digests, DR requirements, and supply-chain policy demanding provenance for everything that runs.

The architecture:

1. One internal registry as the single source of runtime truth (Harbor/Artifactory/ECR/GAR): nothing in production pulls from the internet. Period. Admission policy enforces the registry prefix — this single rule solves rate limits, upstream outages, and unvetted-image risk simultaneously
2. Pull-through proxy cache for upstream (Hub, ghcr, quay): first pull fetches + caches; the fleet hits the cache thereafter. Authenticated upstream credentials (paid Hub account raises limits) held once, centrally — not scattered across nodes. Quarantine mode: new upstream images land in a scan-first zone before general availability
3. Retention as policy, not archaeology: per-repo rules — keep last N tagged + anything deployed (deploy-time labels/annotations feed this), untagged digests GC'd after 14 days, build-cache repos on aggressive cycles. Registries without retention grow ~linearly with CI activity forever; garbage collection windows (some registries need offline/locked GC) scheduled like the maintenance they are
4. Replication: hub-and-spoke — CI pushes to the primary; replication fans out to per-region registries (pull latency + egress cost + regional DR). Kubernetes nodes pull region-local. Failure mode rehearsed: primary registry down = deploys pause but running workloads and node-local cached images are unaffected; regional replica down = fail over pull endpoint
5. The policy surface lives at the registry: image signing verified on push/pull, SBOM attachment required for promotion to the prod/ project, CVE gates per severity tier, robot accounts with per-repo scopes and short-lived tokens (OIDC where supported)

Node-side complement: kubelet serialize-image-pulls=false + sane GC thresholds; pre-pull DaemonSets for giant images before rollout waves; imagePullPolicy: IfNotPresent with digest pins (correctness via digest, speed via cache).

One-liner: 'the registry is tier-1 infrastructure: one internal source of truth, proxy-cached upstream, policy enforced at push and pull, retention automated, and replicas where the pullers are — Docker Hub is a supplier, never a runtime dependency.'

Situation: critical RCE in a ubiquitous library; unknown exposure across ~300 services / ~5,000 running containers.

Phase 1 — scope (first hour): the SBOM payoff. Query the SBOM store (every image built in the last N years has one attached at push): which images contain log4j-core [vulnerable range] → list of image digests → join against what's actually running (cluster inventory of image digests per workload, which you have because deploys record digests) → ranked exposure list in ~30 minutes: internet-facing first, then internal-with-sensitive-data, then batch. Without SBOMs this phase is grep-across-300-repos and takes the weekend — the difference IS the program you built beforehand.

Phase 2 — mitigate before patching (hours 1-4): for internet-facing exposed services, don't wait for rebuilds: WAF rules for the exploit signature (imperfect, buys time), egress lockdown (this class of exploit needs outbound callbacks — default-deny egress turns RCE into a much smaller bug; if you had it already, say so with a small smile), feature flags disabling vulnerable code paths where known, and the env-var/JVM-flag mitigations the advisory offers — rolled via config, not rebuilds (minutes, not hours).

Phase 3 — patch the fleet (hours 4-48): fix lands in ONE place — the golden base or the shared dependency layer — then the rebuild machinery earns its keep: automated PRs bumping the base/dep across affected repos, CI rebuilds (registry cache making them fast), progressive-but-accelerated deploys (canary windows shortened deliberately — the risk calculus inverts when the alternative is running exploitable code). Fleet coverage tracked on a live dashboard: % of exposed digests replaced, by tier. Services that can't rebuild (abandoned, vendor images) get: vendor escalation, runtime isolation (network quarantine, dedicated nodes), or scheduled kill — documented risk acceptance for anything that stays.

Phase 4 — verify + learn (week after): rescan the fleet (running digests vs vulnerable list = zero), hunt for exploitation-in-window (egress logs, Falco/IDS records against IoCs — you had the retention, right?), and the retro focuses on system gaps: which services lacked SBOMs, whose rebuild took >24h and why, where egress-deny wasn't yet rolled out.

The numbers that make it a story: exposure scoped in 30 min, mitigations live in 4h, 92% of exposed fleet rebuilt+deployed inside 48h, stragglers quarantined — versus the industry median of weeks for Log4Shell.

What they're testing: that your answer is 80% pre-built machinery (SBOMs, digest inventory, golden base, rebuild automation, egress posture) and 20% incident execution — heroics are what teams without the machinery call Friday night.

Situation: 200 services, ~40 teams, Dockerfile anarchy: 30+ distinct base images (some 3 years stale), root containers as the norm, build times averaging 15 minutes, no SBOMs, image sizes from 80MB to 4GB for equivalent services, and every security review a bespoke negotiation.

Task: converge the fleet on hardened, fast, observable container practices — without a mandate war, and measurably within three quarters.

Action:

1. Built the paved road first (nothing to adopt = nothing adopted): golden base images per runtime (non-root, tini, CA/tzdata, weekly rebuilds); reference Dockerfiles (multi-stage, cache-optimized, distroless-final where viable) as templates in the service scaffolder; a shared CI build workflow (buildx + registry cache + SBOM + sign + scan) consumable as one include-line
2. Made the right thing the easy thing: new services got all of it by default from the generator; existing services could adopt the CI workflow without touching their Dockerfile (immediate SBOM/signing coverage), then migrate bases at their pace — decoupling the adoption units mattered; all-or-nothing migrations stall
3. Motivated with their pain, not our policy: the pitch per team used their numbers — 'your build is 15 min; the reference cuts it to 4' (cache mounts + ordering), 'your image is 1.8GB; multi-stage gets 200MB — your pods schedule faster'. Speed sold what security couldn't
4. Migration sprints, funded: platform engineers pair-migrated the 20 highest-risk services (internet-facing, stale bases) ourselves — the golden path proven on the hard cases generated the internal case studies that moved the middle 60%
5. Ratchet at the end, not the start: after ~70% organic adoption, admission policies flipped to enforce: corp-registry-only, signature required, non-root required, image age < 30 days for new deploys. Exception path: labeled, expiring, reported — 12 exceptions at flip, 3 a year later

Result: 9 months: 187/200 on golden bases; median build 14min → 3.5min (the number teams actually thanked us for); fleet CVE-critical count down 94% (mostly the weekly-rebuild effect, not heroics); image P50 620MB → 140MB; and the Log4Shell-class drill (we ran one) scoped exposure in 25 minutes via the now-universal SBOMs.

What I'd do differently: start the metrics dashboard on day one — we built adoption tracking in month two and flew blind through the early narrative battles; and version the golden base's contract explicitly (its guaranteed contents) — early silent changes to it broke two teams' assumptions and cost trust we'd carefully built.

What they're testing: platform leadership mechanics — paved road before policy, adoption decoupled and self-interested, enforcement as the ratchet not the spearhead, and metrics that measure outcomes (build time, CVE age, scope-time) rather than compliance checkboxes.

Why it matters: scale-out latency = pull + start + warm; at 2.5 minutes, autoscaling can't chase traffic spikes — you either over-provision (pay) or drop requests (worse). Attack both terms.

Half 1 — the pull (90s):

1. Shrink what moves: multi-stage + slim/distroless base (4GB usually means build tools + test deps shipped to prod — 4GB → 400MB is common); dependency layers ordered for stability so most of the image is already node-cached across deploys (only the app layer changes = only MBs pulled)
2. Cache where you start: imagePullPolicy: IfNotPresent + digest pins; pre-pull DaemonSets stamping hot images onto every node (including fresh nodes via node-startup taints released post-pull); warm node pools sized for spike absorption
3. Stream instead of waiting: lazy-pulling snapshotters — eStargz/SOCI/Nydus: containers start when the needed files arrive, not the whole image (production JVM apps typically touch <20% of image bytes at boot) — pull-to-start drops from 90s to ~10s without shrinking anything
4. Registry locality: per-region/per-AZ replicas or a pull-through cache close to nodes — cross-region pulls are both slow and billed

Half 2 — the warmup (60s JVM):

1. CRaC (Coordinated Restore at Checkpoint): checkpoint the warmed JVM (post-JIT, pools filled) at build/bake time; restore in ~1s at runtime — the JVM equivalent of a hibernate file. Framework support (Spring 6.1+, Micronaut) is production-real now
2. AOT paths: GraalVM native-image (ms startup, no JIT warmup — trade: build complexity, some peak-throughput loss, reflection config) — right for scale-to-zero and spiky autoscaling tiers; keep JIT for steady high-throughput cores
3. Cheaper wins first: AppCDS (class-data sharing — 20-40% startup cut for a build flag), tiered-compilation tuning, lazy-init frameworks (spring.main.lazy-initialization), trimming the classpath (that 60s partly is the 4GB talking)
4. Platform-level masking: startup probes so K8s doesn't kill slow starters + readiness gating with slow-start/LB ramping so warming instances take partial traffic (JIT warms under load safely) — perceived warmup cost drops even where actual warmup remains

The measurement discipline: instrument pull/start/ready/warm as separate spans on a dashboard — teams chronically optimize the wrong term (shaving image MBs when 80% of the latency is JIT, or buying CRaC complexity when the pull was the problem).

Result shape: 4GB/150s → 380MB image, lazy-pull start ~8s, CRaC restore ~2s: scale-out under 15 seconds, autoscaling that actually autoscales, and the over-provisioning buffer cut by half — which is usually what paid for the project.

One-liner: 'cold-start is two bills — bytes moved and JIT earned; streaming snapshotters and checkpoint-restore pay each one down an order of magnitude, but measure which bill is yours before paying either.'

The mechanical truth: containers need a Linux kernel — on macOS/Windows, Docker Desktop runs a hidden Linux VM (Virtualization.framework/WSL2); every container actually runs there. Three consequences drive all the pain:

1. Bind-mount I/O crosses the VM boundary: the host filesystem is shared into the VM (VirtioFS on modern macOS — much better than the osxfs era, still not native): metadata-heavy workloads (node_modules, git status in-container, webpack watching 50K files) run 2-10x slower than Linux-native. This is THE complaint behind 'Docker is slow on Mac'
2. The VM has fixed resources: Docker Desktop's CPU/RAM allocation (not the host's total) is what containers see — the default 2GB/half-cores explains many 'works in CI, OOMs locally' mysteries; file-watching across the boundary also breaks inotify assumptions (hot-reload needs polling fallbacks)
3. Networking is VM networking: localhost in a container ≠ host localhost (hence host.docker.internal); host networking mode historically didn't exist on Mac (recent Desktop versions add a form of it)

The fixes, by leverage:

1. Keep hot I/O out of bind mounts: source code bind-mounted (small, fine), but dependencies and build artifacts in named volumes (node_modules volume overlay — the classic pattern) or built into dev images; watch-mode via polling where inotify fails
2. Mutagen/file-sync modes (Docker Desktop synchronized file shares): two-way sync into the VM instead of live sharing — near-native speed for the brutal cases; costs sync lag and disk duplication
3. Move the dev environment into the VM/container entirely: devcontainers — the editor attaches remotely, code lives on the Linux side, bind-mount pain disappears; also standardizes toolchains (its own win)
4. Consider the alternatives where licensing/perf demand: Colima/Lima, Rancher Desktop, Podman Desktop, OrbStack (notably faster VM + FS layer) — evaluate against Desktop's licensing cost for orgs >250 seats since that's a real budget line
5. Resource + hygiene defaults shipped to the team: a documented Desktop config (6+ CPUs, 8-12GB, VirtioFS on), docker system prune schedules (Desktop VMs quietly grow 60GB disk images), and the compose override that implements the volume patterns above — DX debt is a platform problem; every developer solving it alone is 40 bad solutions

One-liner: 'on Mac and Windows every container lives in a hidden Linux VM — the perf story is entirely about what crosses that boundary; keep hot files on the Linux side (volumes, sync, or devcontainers) and Docker Desktop stops being the team villain.'

The reality check first: compose-in-prod is more common than conference talks admit — single-VM deployments with docker compose up -d and a systemd unit. It works until it doesn't: no multi-node, no self-healing beyond restart, deploys are down && up outages, secrets are .env files.

What maps cleanly (the mechanical 60%):

• services: → Deployments + Services (service names already match K8s DNS conventions if compose networking was used idiomatically)
• environment: → env/ConfigMaps; ports: → Services/Ingress; healthcheck: → probes (semantics transfer directly); restart: → implicit in Deployments; resource limits: → resources
• Kompose generates a first draft — useful as scaffolding, never as the destination (its output is literal, not idiomatic)

What doesn't map — the design work:

1. depends_on ordering dies: K8s has no start-ordering between Deployments — apps must retry dependencies at startup (they should have anyway; compose let teams skip building it). This surfaces as 'app crashes on cluster restart' and the fix is app-level resilience or initContainer gates
2. Volumes: compose named volumes on one host → PVC decisions (storage classes, access modes, backup) — and the buried question 'wait, is this stateful?' for things like the postgres container that was always just... there. Honest answer is often 'move it to RDS during the migration, not after'
3. .env files → real secret management (ESO/Vault) — the migration is the forcing function for the secrets hygiene that never happened
4. Build-on-deploy (build: in prod compose) → registry pipeline — images must become versioned artifacts with CI; another overdue forcing function
5. Host-coupled assumptions: bind mounts to host paths, network_mode: host, hardcoded localhost ports, container names as identity — each is an app change, and finding them all is why you inventory before you lift

Sequencing (strangler, not big-bang):

1. Pre-work on the compose stack itself: images to a registry with tags, secrets out of .env, retry-on-startup into apps, healthchecks everywhere — each lands independently and de-risks the move while improving current prod
2. Stand up the cluster + platform substrate (ingress, ESO, observability) with the golden-path tooling — the migration inherits standards instead of porting anarchy
3. Move stateless leaf services first (traffic-shifted via DNS/LB weights, compose stack still serving as fallback), data tier last or never (→ managed services)
4. Run hybrid deliberately for weeks: same observability on both sides, cutover per service with rollback = weight-flip back
5. Decommission ceremony: the compose VM read-only for a month before deletion — the forgotten cron on that box is a law of nature

One-liner: 'the YAML translation is a weekend; the real migration is paying down what compose let you defer — startup resilience, secret hygiene, artifact discipline, and the stateful question — pay it before the move and the move itself is boring.'

The contract: containers log to stdout/stderr, unbuffered, structured (JSON) — the runtime captures the streams; shipping is the platform's job, not the app's. This inversion (vs apps managing log files/rotation/shipping) is the whole design: apps stay ignorant of destinations; the platform changes them without app releases.

Driver mechanics (Docker): default json-file writes per-container files under /var/lib/docker/containers/ — unrotated by default: the 40GB log file that fills the disk is a rite of passage; daemon.json max-size/max-file is day-one configuration, not an optimization. Alternatives: local (better compression/format, same node-local model), direct-ship drivers (fluentd, awslogs, gelf...) — with the classic trap that blocking-mode direct drivers couple app liveness to log-endpoint liveness (endpoint slow → writes block → app stalls: a logging outage becomes an app outage); non-blocking mode + ring buffer trades that for silent drops. Node-local files + an independent shipper is the resilient default.

The Kubernetes shape: kubelet handles capture/rotation (containerLogMaxSize); a DaemonSet shipper (Fluent Bit typically) tails node files, enriches with pod metadata (namespace/labels/pod — the k8s filter), and ships to the backend. App → stdout → node file → tailer → pipeline: each link independently buffered and monitorable.

What breaks at scale (the checklist of scars):

1. Multiline — stack traces shredded into 40 one-line events: multiline parsers or (better) JSON logging so the trace is one field
2. Cardinality/volume economics — a debug-level service at 10K lines/sec is a five-figure monthly line item; per-team volume quotas, level enforcement in admission (no DEBUG in prod), and sampling on high-volume INFO paths
3. Backpressure — backend outage → shipper buffers → node disk pressure → now it's a scheduling incident: bounded buffers with drop policies (drop DEBUG before INFO before ERROR — explicitly configured, not discovered), and shipper lag as a paged metric
4. The shipper is production software: its resource limits (it OOMs; its CPU throttling = silent log lag), its version upgrades, its parse-error rate — all monitored like any tier-1 service; 'logs were missing for 3 days' post-incidents trace here depressingly often
5. Enrichment discipline: trace_id in every line (log↔trace correlation is where debugging speed lives), consistent schema org-wide (a level field that's sometimes severity breaks every dashboard)

One-liner: 'apps write structured lines to stdout and know nothing else; the platform owns capture, rotation, enrichment, shipping, and the economics — and the shipper pipeline is tier-1 software with its own failure modes, not plumbing you configure once.'

Why devices break the container abstraction: containers virtualize CPU/memory/filesystem via kernel primitives — but a GPU is a device: driver stacks, device nodes (/dev/nvidia*), and userspace libraries that must match the host kernel driver version. The clean 'image runs anywhere' story now has a host-coupling seam.

The mechanics (NVIDIA as the canonical case):

• Host: kernel driver installed on the node (node image/DaemonSet-managed) — never in the container
• Container: CUDA userspace libs in the image (the nvidia/cuda base tiers: base/runtime/devel), compatible with the host driver (CUDA forward-compat rules — the version matrix that generates half of all GPU tickets)
• The bridge: NVIDIA Container Toolkit — injects device nodes + driver libs at container start; in K8s, the device plugin advertises nvidia.com/gpu: 1 as a schedulable resource; RuntimeClass wires the runtime hooks
• Sharing models (GPUs don't oversubscribe like CPU): exclusive-per-pod (default — expensive for small inference), MIG (hardware partitioning on A100/H100-class: real isolation, fixed-size slices), time-slicing (soft sharing — no memory isolation, noisy-neighbor real), MPS — pick per workload class; a fleet mixing training and small-inference needs at least two of these deliberately

What changes operationally:

1. Scheduling economics dominate: GPU nodes are 10-50x CPU node cost — bin-packing, queueing (Kueue/Volcano for batch/training gangs), and utilization dashboards per-GPU (not per-node) are where the money lives; idle-GPU% is the platform KPI
2. Image size explodes: CUDA runtime images are 2-6GB — the pull-time work (lazy-pull snapshotters, pre-pull, regional caches) stops being optional
3. Node lifecycle couples to drivers: driver upgrades = node pool rebuilds coordinated with workload compatibility matrices; the GPU node pool is its own upgrade train, tainted and version-labeled
4. Health is device-level: DCGM exporter → per-GPU metrics (memory, ECC errors, thermal throttling); a 'Ready' node with a sick GPU needs node-problem-detector rules to cordon — kubelet doesn't know

The ML platform framing: training (gang-scheduled, checkpointed, spot-tolerant batch — queue-managed, preemptible) and inference (latency-bound serving — right-sized slices, autoscaled on QPS/latency, MIG/time-sliced) are different platforms sharing hardware; conflating them wastes the most expensive compute you own. Data movement (dataset volumes, model registries, cache layers) usually gates throughput before FLOPs do.

One-liner: 'GPU containers re-couple you to the host — driver matrices, device plugins, and sharing models are the new complexity — and the platform job shifts from "schedule the pods" to "never let a $40K card sit idle while data loads".'

The kernel mechanics (cgroups doing the work):

• --memory=512m → cgroup memory limit: exceed it and the OOM killer terminates the biggest offender in the cgroup (exit 137). Memory is incompressible — there's no throttling, only death. --memory-swap controls swap participation (usually disabled in orchestrated environments)
• --cpus=2 → CFS quota/period (200ms quota per 100ms period): a hard ceiling — hit it and the process is throttled (paused until the next period), even if host cores sit idle. CPU is compressible — you slow down, you don't die
• --cpu-shares=512 → proportional weight only under contention: an idle host lets any container burst; a contended host divides CPU by share ratio. Shares ≠ limits — the perennial confusion

The observable consequences:

1. Throttling is the silent latency killer — CPU graphs look fine (say, 40% average) while nr_throttled/throttled_time (cpu.stat) climbs: the app burns its quota in bursts and stalls tail latencies. Multi-threaded runtimes hit this hard (8 threads × 25ms = 200ms quota gone in 25ms of wall time)
2. OOM kills vs memory leaks: cgroup memory includes page cache — 'memory keeps growing' is often cache (reclaimable, fine), not a leak; container_memory_working_set_bytes is what the killer effectively judges

Right-sizing method (not vibes):

1. Load-test with production-shaped traffic; record P99 memory and CPU burst patterns (not averages)
2. Memory: P99 + 20-30% headroom, runtime configured to respect it (JVM MaxRAMPercentage ~70)
3. CPU: set based on latency sensitivity — latency-critical services get limits high enough that throttling is ~zero (monitor cpu.stat, alert on throttle ratio), or follow the K8s school of 'requests always, CPU limits rarely'
4. Re-measure quarterly and after major releases — right-sizing decays

One-liner: 'memory limits kill, CPU limits throttle, shares only matter under contention — monitor OOM events and throttle time, not utilization averages, and size from measured P99 bursts.'

The mechanism people never think about: docker build . doesn't let the builder reach into your filesystem — it tars the entire context directory and ships it to the daemon/builder first. Every file, whether any COPY references it or not. That 'Sending build context to Docker daemon 2.3GB' line is the tax being paid.

Why it's worse than slow:

1. Speed: 2.3GB tarred and transferred before the first instruction runs — on remote builders (CI, buildx remote) that's network transfer, every build
2. Cache poisoning: COPY . . fingerprints the copied fileset — a .git directory (changes on every commit by definition), editor swap files, or logs in the context invalidate the layer even when no source changed: 'why did my build not cache' is this, constantly
3. Secret leakage: .env, credentials files, SSH keys sitting in the project root get COPY'd by broad COPY . . into image layers — permanent (layer history!), pushed to registries, found by scanners later. Build-context hygiene is a security control

.dockerignore design (deny-first for serious repos):

# Start from everything-denied for tight control:
*
!src/
!package.json
!package-lock.json
!tsconfig.json

Allow-listing (* then ! exceptions) inverts the default: new junk in the repo is excluded automatically instead of remembered manually. Minimum-viable deny-list version: .git, node_modules (rebuilt inside anyway), dist/build, *.log, .env*, docs, tests if not built-in-image.

The related BuildKit upgrades: context is sent incrementally/on-demand with BuildKit (better but not free), --mount=type=bind can access build files without COPYing them into layers, and remote contexts (docker build https://github.com/...) skip local state entirely.

Verification habit: docker build output's context size line belongs in code review consciousness — a jump from 40MB to 900MB means someone committed something that doesn't belong, and the image probably now contains it.

One-liner: 'the context is an implicit COPY of your whole directory to the builder — .dockerignore is both your build-speed budget and a leak barrier; allow-list it and watch the size line like a metric.'

The policies, precisely:

• no — dead stays dead (default). For: one-shot tasks, CI steps, anything a supervisor above Docker owns
• on-failure[:max] — restart only on non-zero exit, optionally capped: 'this should succeed; retry N times then give up'. The only policy with a terminal failure state — which is exactly what batch jobs need
• always — restart regardless, including after daemon/host reboot — even if you docker stopped it, a daemon restart resurrects it (the surprise: stopped-for-maintenance containers coming back at 3am after a host patch)
• unless-stopped — always minus the resurrection: manual stops are respected across daemon restarts. The right default for compose-managed daemons on VMs

Backoff behavior: Docker restarts with exponential delay (100ms doubling, capped) but — critically — resets the backoff after a container runs 10s+: an app that limps for 11 seconds then dies restarts forever with no meaningful backoff, hammering downstream dependencies (the thundering-herd-against-the-database failure mode). Kubernetes' CrashLoopBackOff (capped at 5min, slower reset) is more conservative for the same reason.

Designing crash behavior (the senior layer):

1. Crash-only design for daemons: the app should be safe to kill at any instant (journaled state, idempotent startup, no 'clean shutdown required') — then unless-stopped/orchestrator restarts are always-safe medicine. If restarts can corrupt state, the restart policy isn't your problem
2. Fail fast at startup, not endlessly retry inside: an app that can't reach its config/DB should exit non-zero (letting restart policy + backoff own the retry cadence and making the failure visible as restart counts) rather than log-and-spin internally where nothing counts it — with the balance point that dependency blips shouldn't kill a running daemon (retry in-app once warm; exit only when startup prerequisites are absent)
3. Jobs: on-failure:3 + alerting on the terminal failure, never always (a doomed job on always is an infinite loop with a fan)
4. Observability contract: restart count is a first-class alert signal (docker events/RestartCount inspect field; K8s restart metrics) — a service 'up' with 40 restarts/hour is an incident wearing an SLA-green costume

One-liner: 'restart policies are supervision contracts — match them to workload semantics (daemons: unless-stopped; jobs: on-failure with a cap), design apps crash-safe so restarts are always-medicine, and treat restart count as the health metric uptime hides.'

The three channels and their physics:

1. Baked into the image (COPY config.yaml, ENV in Dockerfile): immutable, versioned with the artifact, identical everywhere the digest runs — which is precisely its failure: config changes require rebuilds, and per-environment values violate build-once-promote-everywhere. Right for: true constants (framework mode flags), sane defaults meant to be overridden
2. Environment variables at runtime: the 12-factor default — injected per environment, no rebuild, universally supported. The trade-offs people under-weight:
   • Leak surface: env is visible in docker inspect, /proc/<pid>/environ, often dumped wholesale into error reports/crash telemetry, and inherited by every child process (that shell-out to a vendor CLI just got your DB password)
   • No rotation: env is fixed at process start — a rotated credential requires restart; no hot reload
   • Flat string namespace: nested/structured config gets encoded awkwardly (JSON-in-env), typos fail silently
3. Mounted config (files via volumes; K8s ConfigMaps/Secrets as mounts): structured, hot-reloadable (mounted Secrets update in-place — apps that watch files can rotate without restart), not in process env (smaller leak surface, not inherited by children), auditable as objects. Costs: app must read files (trivial), and mount propagation timing (K8s: up to a minute for updates)

The hierarchy I give teams:

• Secrets → mounted files, always (or runtime fetch from a secrets manager with workload identity — the gold standard: short-lived, rotated, audited): env vars for secrets is the pattern every security review flags and every breach retro regrets
• Environment-shape config (endpoints, feature flags, tuning) → env vars for simple values, mounted config files for structured — sourced from the platform (ConfigMaps, launch config), never baked
• Behavioral constants → baked with runtime override capability (ENV defaults in Dockerfile that env can override — defense in depth of defaults)
• Precedence must be explicit and documented: baked defaults < mounted file < env var < flags — apps that read all four with unclear precedence generate un-debuggable environment drift

The 'same image everywhere' test: if you can't run the exact prod digest in staging by only changing injected config, the boundary between artifact and configuration is broken somewhere — find it before the promote-a-hotfix-at-2am moment finds it for you.

One-liner: 'bake defaults, inject environment-config, mount secrets — env vars are convenient but leak and can't rotate, so their contents should be things you could shout across the office.'

The architectural difference (not just branding): Docker runs a root daemon (dockerd) that all CLI commands talk to — every container is a child of that daemon, and socket access = root. Podman is daemonless: the CLI directly forks containers (via conmon per container), no central process, no socket-shaped root backdoor; containers are children of your shell/systemd session, visible in your process tree, auditable as you.

What that buys, concretely:

1. Rootless as the default posture, done deeper: Podman pioneered production-grade rootless (user namespaces mapping container root → your UID; networking via slirp4netns/pasta) — a container escape lands as an unprivileged user, not root. Docker has rootless mode too now; Podman's is the paved path rather than the alternate one
2. The fork-bomb of consequences from 'no daemon': no single point of failure/upgrade for all containers (daemon restart ≠ container restart concerns), per-user isolation on shared hosts (each user's containers in their own space — multi-tenant build hosts, HPC login nodes), and systemd-native operation (Quadlet: containers as systemd units with dependencies, restart policies, and journal logging — the natural fit for VM/edge deployments where systemd already runs the host)
3. Auditability: actions attribute to real UIDs in the process tree and audit logs — 'which human started this container' has an answer without daemon-log archaeology

Compatibility reality: same OCI images, same registries, CLI-compatible (alias docker=podman mostly works), Dockerfiles build via buildah under the hood; Compose support exists (podman-compose / socket-compat mode) but is the roughest edge — heavy Compose-based dev workflows should validate before switching. Kubernetes doesn't care (it runs containerd/CRI-O anyway — node runtime was never Docker-the-daemon after dockershim's removal).

When it matters organizationally:

• CI runners and shared build infrastructure — the strongest case: build jobs execute semi-trusted code; daemonless+rootless removes the socket-mount/dind root-exposure class entirely
• RHEL-ecosystem shops: Podman is the first-class citizen (RHEL ships it; Docker is the add-on) — fighting the distro default has ongoing costs
• Edge/VM fleets managed by systemd — Quadlet beats bolting Docker restart policies onto systemd
• Docker Desktop licensing ($ per seat >250 employees) makes Podman Desktop a real line-item conversation
• Where it doesn't matter: K8s production nodes (neither runs there), and teams deeply invested in Docker Desktop/Compose ergonomics where the migration friction outweighs the posture gain

One-liner: 'Podman removes the root daemon from the trust equation — same images, same Dockerfiles, but containers run as you under systemd; it matters most where untrusted code builds (CI) and where systemd already governs (edge), not on your K8s nodes where neither tool lives anymore.'

The bind: hardened images (distroless, no shell, read-only, non-root) are correct — and undebugable by 2015 habits (exec -it bash finds no bash). The answer is attaching tooling at debug time instead of shipping it always.

The technique ladder:

1. Namespace-sharing toolbox (Docker): docker run -it --rm --network container:<target> --pid container:<target> nicolaka/netshoot — a full toolkit container inside the target's network and PID view: its ports are your localhost, its processes visible; tcpdump, ss, curl, strace from the sidecar while the target stays pristine
2. Kubernetes ephemeral containers (the first-class version): kubectl debug -it pod/api --image=netshoot --target=app — injects a debug container into the running pod (shares namespaces with --target for process visibility); no restart, no spec change, RBAC-gated, audit-logged. This is the front door for prod debugging now
3. Node-level when pod-level isn't enough: kubectl debug node/<node> -it --image=... (host namespaces via a privileged debug pod) for kernel/runtime-layer issues — conntrack, cgroup stats, containerd state
4. Static-binary drops for the stubborn cases: kubectl cp a statically-linked tool (busybox-static, or a purpose-built debug binary) into a writable volume path — works even where ephemeral containers are unavailable (old clusters, some managed runtimes); read-only rootfs means you need some writable mount, which your tmpfs mounts provide
5. The observability alternative to shells entirely: language-level (continuous profiler, heap dumps via signal/endpoint) and eBPF-based tooling (Pixie-style: syscall/network/profiling visibility with zero in-container anything) — increasingly the answer that scales past 'ssh-with-extra-steps'

Filesystem forensics without exec: kubectl debug --copy-to (duplicate the pod with an added shell for offline poking), docker commit + run-with-shell on the corpse for Docker, or crictl/ctr snapshots at the node — examine the patient's filesystem without operating on the live patient.

The governance layer (senior signal): prod debug access is designed, not improvised — ephemeral-container RBAC scoped to an on-call role, debug images from an internal allowlisted registry (a random Hub toolbox in prod is supply-chain risk mid-incident), session TTLs and audit review, and the target container's security context still applies (debug sidecars don't get privileged just because it's an emergency — that's a separate, logged break-glass).

One-liner: 'hardened images moved debugging from "tools inside" to "tools attached" — namespace-sharing sidecars and kubectl debug give you everything exec gave, on demand, audited — and the endgame is eBPF observability that needs no attachment at all.'

The invariant everything hangs on: build ONCE, promote the digest. Rebuilding 'the same' image per environment produces different artifacts (timestamps, dependency drift) — whatever staging validated, prod wouldn't be running it.

Stage 1 — Build (one pipeline, one output): buildx (rootless builder, registry cache) → push by digest to the staging/ project; attached at push: SBOM (syft), provenance attestation (SLSA — repo, commit, workflow), cosign keyless signature bound to the CI identity. Unit + integration tests precede push; the digest is the pipeline's sole output contract.

Stage 2 — Dev/auto-staging deployment: GitOps repo PR (automated) pins the new digest for dev; auto-merge on green. Staging promotion adds: scan gate (fail: critical-with-fix; warn+ticket: high), smoke + contract tests against the deployed environment, and a soak window (hours to a day, error-budget-watched).

Stage 3 — Prod promotion: a PR moving the digest pin into prod values — the diff IS the review (one line, one digest, linked evidence: staging soak dashboard, scan report, provenance). Human approval for tier-1 services; automated for low-tier with green gates. Registry-side promotion: the digest is re-tagged/replicated into the prod/ project — which policy-requires signature + fresh scan attestation to accept.

Stage 4 — Deploy + verify: progressive rollout (canary %, metric-gated auto-promotion), admission control verifying signature/registry/provenance at schedule time (the runtime enforcement that makes the pipeline unbypassable — API-applied YAML with a rogue image simply won't schedule).

Rollback design (decided before it's needed): previous digests stay in the registry under retention pinned-by-deployment-history; rollback = revert the GitOps digest PR (one line back) — no rebuild, no re-approval queue (pre-authorized paved path for reverts), canary-aware (rollback also rolls progressively unless severity says slam it). The rollback drill is rehearsed: target <5 min from decision to traffic-on-old-digest.

The audit answer (what regulators/security actually ask): for any running pod: image digest → signature (who built) → provenance (from what commit) → SBOM (containing what) → scan history (known-vulnerable when?) → deployment PR (who approved) — a complete chain queryable in minutes, because every link was attached at build time, not reconstructed later.

One-liner: 'the pipeline is a digest acquiring evidence as it moves — tests, scans, signatures, soak time, approvals — with environments as pointers to it and admission control as the enforcement; rollback is just pointing back at yesterday's evidence.'

What a signature proves (be precise — this is the question inside the question): that a specific digest was attested by a specific identity at a time — nothing more. It does NOT prove the image is safe, bug-free, or well-built; it proves provenance: 'our CI pipeline, from this repo/workflow, produced these exact bytes.' Security value comes from the policy you attach to verified identities.

The legacy: Docker Content Trust (Notary v1) — TUF-based tag signing, DOCKER_CONTENT_TRUST=1. Architecturally sound, practically dead: key management UX drove adoption to ~zero, tag-oriented signing fit registries awkwardly. Know it as history; don't build on it.

The current answer: sigstore/cosign:

• Keyless signing: the CI job authenticates via OIDC (GitHub Actions/GitLab workload identity) → Fulcio issues a short-lived cert binding the identity (repo:corp/payments:ref:refs/heads/main + workflow) → signature over the image digest stored in the registry as an OCI artifact → transparency log entry in Rekor (public, append-only — signing events are auditable, and a stolen signing capability leaves footprints). No long-lived keys to manage or leak — the property that made this adoptable where DCT wasn't
• Attestations beyond the bare signature: in-toto/SLSA provenance (how it was built: source repo, commit, builder), SBOM attestations, vulnerability-scan attestations — each signed, each attached to the digest; 'the image' becomes a verifiable evidence bundle

Verification — where the value materializes: admission control (Kyverno verifyImages / sigstore policy-controller) enforcing per-namespace policy: prod namespaces require signatures from the prod pipeline identity on the exact digest, plus a fresh scan attestation. Deploy-time verification in CD as an earlier gate. Unverifiable image = unschedulable, which converts supply-chain policy from documentation into physics.

Design decisions you'll defend in the interview:

1. Sign digests, never tags — tags move; the signature must bind to content
2. Identity-based (keyless) over key-based for CI — key custody is the failure mode; workload identity rotates itself. Keep key-based (KMS-held) for the rare human-signed exception path
3. Policy granularity: per-environment signer requirements (dev: any corp pipeline; prod: the release workflow identity specifically) — the signature scheme is only as strong as the weakest identity your policy accepts
4. Air-gapped/regulated variants: private Rekor/Fulcio or KMS-key signing — same verification model, self-hosted trust roots

One-liner: 'signing binds bytes to identity; sigstore made the identity part operationally free (OIDC instead of key custody), and admission policy turns it into an actual control — the mature stack signs digests keylessly in CI, attaches SBOM+provenance attestations, and lets prod schedule nothing it can't verify.'