Container Fundamentals

TL;DR

• OCI (Open Container Initiative) standardizes image format and runtime interface; this is why Docker-built images run on containerd/CRI-O/Podman without conversion.
• Layers are cached and content-addressed; unchanged layers reuse cache, and identical layers are stored once across images, which is why Dockerfile instruction order (expensive steps before cheap ones) matters.
• Mutable tags (latest) can point to different images over time, breaking reproducibility; immutable digests guarantee the same bytes every time.
• Registry outages are a real risk if you depend entirely on public registries; pull-through caches or private mirrors remove that critical dependency.
• Instruction ordering is the highest-leverage Dockerfile optimization: putting dependency installation before source code copy keeps dependencies cached across most builds.

Core Concepts

OCI: the interoperability standard

OCI (Open Container Initiative) defines two key specs:

• Image spec: the format and structure of a container image
• Runtime spec: how a runtime executes a container

This standardization decouples “what built this” from “what can run it.” A Docker-built image, a Podman-built image, or a BuildKit-built image are all OCI-compliant and can be run by any OCI-compliant runtime (Docker, containerd, CRI-O, Podman, rkt). No vendor lock-in, no conversion steps, no proprietary format wars — just a common standard everyone implements.

Layers: the foundation of efficient builds

A Docker image is a stack of read-only layers, each corresponding to a Dockerfile instruction that changes the filesystem (RUN, COPY, ADD). Layers are:

• Content-addressed: each layer has a hash of its content
• Cached: an unchanged layer from a previous build is reused, not rebuilt
• Deduplicated: identical layers shared across different images are stored once in the registry

This has direct, measurable consequences:

Instruction ordering matters: Dockerfile instructions run in order. Each instruction invalidates the cache for all subsequent instructions. If source code is copied before dependencies are installed, every source code change invalidates the cache from that point forward, forcing (expensive) dependency installation to re-run:

# ❌ Bad ordering — invalidates cache on every source change
FROM python:3.11
WORKDIR /app
COPY . .                    # Source code (changes frequently)
RUN pip install -r requirements.txt  # Dependencies (expensive, re-runs every time)

# ✅ Good ordering — keeps dependency layer cached
FROM python:3.11
WORKDIR /app
COPY requirements.txt .     # Manifest (changes rarely)
RUN pip install -r requirements.txt  # Dependencies (stays cached)
COPY . .                    # Source code (changes frequently)

The source code layer changes frequently but is cheap. The dependency layer changes rarely but is expensive. Cache optimization saves time on most builds.

Registries and external dependencies

Registries store and serve images. A real operational gap: depending entirely on a single external public registry with no local mirror or pull-through cache.

Registry outage → can’t pull images → can’t deploy → production incident. This is a critical external dependency under circumstances you don’t control.

Mitigation: private registry mirror or pull-through cache for commonly-used base images. Cheap insurance against an outage that isn’t your responsibility.

Tags vs. digests: reproducibility

A tag (myapp:latest, myapp:v2.1) is a mutable pointer. It can be re-pushed to point at an entirely different image at any time:

# Day 1
docker tag myapp:abc123 myapp:latest
docker push myapp:latest

# Day 5
docker tag myapp:xyz789 myapp:latest
docker push myapp:latest    # Same tag, different image

# Two deployments both pulling "myapp:latest" days apart
# Pull different content

A digest (myapp@sha256:abc123...) is content-addressed and immutable. It guarantees the exact same bytes every time, ensuring reproducibility.

Production pipelines that care about reproducibility:

• Pin to an immutable digest, or
• Pin to a genuinely immutable version tag (never re-pushed)
• NOT a floating tag like latest or stable

Image Building and Hosting

Dockerfile Best Practices

# Good practices for layer optimization and reproducibility
FROM python:3.11-slim

LABEL maintainer="devops@example.com"

# Dependency manifest (changes rarely)
COPY requirements.txt /app/requirements.txt
RUN pip install --no-cache-dir -r /app/requirements.txt

# Source code (changes frequently, won't invalidate dependency layer)
WORKDIR /app
COPY . .

# Non-root user for security
USER 1000

# Immutable reference for reproducibility
ENTRYPOINT ["python", "app.py"]

# Build with explicit cache control
docker build \
  --cache-from myregistry.com/myapp:latest \
  --tag myapp:sha256-abc123 \
  --tag myapp:latest \
  .

# Push with immutable reference
docker push myapp@sha256:abc123

Common Pitfalls

• Pitfall: Dockerfile copies source code before installing dependencies; every code change triggers full dependency reinstall. Why: Instruction order wasn’t optimized for cache reuse. Fix: Copy manifest files first (requirements.txt), install dependencies, then copy source code.

• Pitfall: Registry outage during peak deployment; can’t pull base images; production incident. Why: Single external registry dependency with no mirror. Fix: Implement pull-through cache or private mirror for commonly-used base images.

• Pitfall: Two “identical” deployments pull different image content; different behavior or bugs. Why: Both referenced mutable latest tag; tag was re-pushed with different content between deploys. Fix: Pin to immutable digest or genuinely immutable version tag.

• Pitfall: Assumed Docker images need special handling to run on containerd; spent effort on “compatibility layers.” Why: OCI compliance wasn’t understood. Fix: Recognize OCI as the standard; any OCI-compliant image works on any OCI-compliant runtime without conversion.

Interview Questions

• Explain mechanically why a Dockerfile’s instruction order affects build speed and cache efficiency. — Tests understanding of how layer caching works, not just “put things in the right order.”

• Two deployments both referenced myapp:latest on different days but pulled different images. What happened, and how do you prevent it? — Tests understanding of mutable tags versus immutable digests.

• Why can a Docker-built image run on a Kubernetes cluster using containerd without any special compatibility layer? — Tests whether OCI’s role as the standard is genuinely understood.

• Design a Dockerfile for a Python app that optimizes for both build cache reuse and runtime security. — Tests understanding of layer ordering, non-root execution, and image optimization.

• A base image registry goes down during peak deployment. How do you architect around this? — Tests understanding of registry dependencies and mitigation strategies.