19. Image Optimization

TL;DR

• Smaller images = smaller attack surface; the fastest fix for “40 CVEs” is often removing unused packages entirely, not patching 40 things.
• BuildKit cache export/import (--cache-to/--cache-from) persists layer cache across ephemeral CI runners; without it, every build rebuilds every layer from scratch.
• Distroless bases remove shell, package manager, and utilities; you lose interactive debugging but gain security by removing tools attackers could use if they escape.
• Vulnerability scanning in CI (blocking builds) prevents vulnerable images from ever reaching production; scanning post-deployment is cleanup, not prevention.
• Base image version pinning ensures reproducible builds; latest is a moving target that silently pulls different images over time.

Core Concepts

BuildKit and cache persistence

BuildKit is Docker’s modern build engine. Its most operationally valuable feature for CI is cache export/import — persisting layer cache to a registry or object storage so ephemeral CI runners benefit from previously-built layers:

Without cache persistence:

• Fresh CI runner starts with zero cache
• Every layer rebuilds from scratch
• 5-minute build becomes 20 minutes
• Waste across all builds

With cache persistence (--cache-to registry, --cache-from registry):

• Fresh runner pulls cache metadata from registry
• Unchanged layers are skipped
• Build stays fast even on ephemeral runners

# Dockerfile
FROM node:20
COPY package.json /app/
RUN npm install        # Expensive step — 3 minutes
COPY . /app
RUN npm run build      # Expensive step — 5 minutes

# CI build with cache persistence
docker buildx build \
  --cache-from type=registry,ref=myregistry.com/myapp:buildcache \
  --cache-to type=registry,ref=myregistry.com/myapp:buildcache,mode=max \
  --push \
  -t myregistry.com/myapp:latest \
  .

Distroless: removing unnecessary attack surface

Distroless base images contain only what’s needed to run the application: language runtime and the app. Everything else is stripped:

• No shell (/bin/bash)
• No package manager (apt, yum)
• No general utilities (curl, grep, sed)
• No compiler or build tools

Trade-off: you lose the ability to docker exec into a running container and debug interactively. This is intentional. Those tools are exactly what an attacker with code execution would use to explore, escalate privileges, or pivot. Removing them removes the attacker’s toolkit.

# ❌ Full OS base — large, high attack surface
FROM ubuntu:24.04
RUN apt-get update && apt-get install -y curl, wget, vim, etc.
COPY myapp /app
ENTRYPOINT ["/app"]

# ✅ Distroless — minimal, low attack surface
FROM gcr.io/distroless/base-debian12
COPY myapp /app
ENTRYPOINT ["/app"]

Debugging strategy shifts: rely on structured logging, ephemeral debug containers/sidecars, or dump core files for offline debugging — not shelling into production.

Structural security vs. patching CVEs

Image with 40 CVEs in the base. Two approaches:

Patching (symptom treatment):

New CVE found in unused package X → patch it
New CVE found in package Y that app doesn't use → patch it
Repeat 40 times

Still hundreds of packages; still vulnerable to unknown CVEs.

Structural reduction (disease removal):

Switch to distroless base that doesn't include the 40 problematic packages
Removes those 40 CVEs AND all future CVEs in packages never needed
Smallest attack surface possible

The structural fix is higher-leverage: removes the root cause, not symptoms.

Vulnerability scanning timing: prevention vs. cleanup

Post-deployment scanning (cleanup):

• Image already built and pushed
• Image already deployed and running
• Finding a critical CVE means stopping a running service
• Prevention has already failed

Build-time scanning (prevention):

• Scan during CI build
• Fail the build if scanning fails
• Vulnerable image never reaches production
• Prevention before deployment

Prevention (scanning in CI) is far cheaper than cleanup (incident response for a vulnerable image in production).

Base image version pinning

FROM node:latest is a moving target:

# Month 1
FROM node:latest  # Pulls node:20.10

# Month 6, rebuild same Dockerfile
FROM node:latest  # Pulls node:21.0 (different major version!)

Same Dockerfile, different behavior over time. Breaking changes, different package versions, new vulnerabilities.

Pinned version:

FROM node:20.10-alpine          # Specific version tag
# or
FROM node:20.10@sha256:abc123   # Specific digest (most reproducible)

Same Dockerfile, same result today and six months from now.

Image Optimization Strategies

Configuration Reference

# BuildKit with cache persistence (GitHub Actions example)
name: Build
on: [push]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: docker/setup-buildx-action@v2
      - uses: docker/build-push-action@v4
        with:
          push: true
          cache-from: type=gha
          cache-to: type=gha,mode=max
          tags: myapp:latest
          context: .

# Distroless Dockerfile
FROM golang:1.21 AS builder
WORKDIR /src
COPY . .
RUN go build -o myapp

FROM gcr.io/distroless/base-debian12
COPY --from=builder /src/myapp /app
USER 1000
ENTRYPOINT ["/app"]

# Version pinning in Dockerfile
FROM node:20.10-alpine@sha256:abc123def456
# or minimum:
FROM node:20.10-alpine

# Vulnerability scanning in CI (Trivy example)
trivy image --exit-code 1 --severity HIGH,CRITICAL myapp:latest

Common Pitfalls

• Pitfall: CI pipeline rebuilds every layer from scratch on every run, taking 20 minutes when it should take 2. Why: BuildKit cache export/import was never configured. Fix: Enable cache persistence (--cache-to registry); ephemeral runners can then benefit from cached layers.

• Pitfall: Base image has 40 CVEs; months spent patching them individually. Why: Structural reduction (switching to a smaller base) wasn’t considered. Fix: Ask first: “do all these packages need to be in the image?” Usually the answer is no; switch to a minimal or distroless base.

• Pitfall: Critical CVE discovered in a production image that’s been running for weeks. Why: Vulnerability scanning only happened post-deployment. Fix: Scan in CI, fail the build on vulnerabilities; prevention before production.

• Pitfall: Dockerfile uses FROM node:latest; rebuild months later produces different behavior due to silent major version change. Why: Version wasn’t pinned. Fix: Pin to specific version tag, ideally a digest, for deterministic builds.

• Pitfall: Team loses ability to debug running containers after switching to distroless; thinks this is a weakness. Why: Debugging strategy wasn’t updated. Fix: Plan logging, metrics, and ephemeral debug containers/sidecars as the debugging approach; shelling into production is an antipattern anyway.

Interview Questions

• A base image has 40 CVEs, mostly in packages the app never uses. What’s your remediation strategy? — Tests whether structural base-image reduction is understood as the highest-leverage fix.

• Explain the security trade-off of distroless base images. — Tests understanding that loss of shell is an intentional security benefit, not a regrettable side effect.

• Your CI pipeline takes 20 minutes to build an image that changes 5 lines of source code. What’s the problem, and how do you fix it? — Tests understanding of cache persistence and BuildKit optimization.

• Why should vulnerability scanning happen in CI rather than only against a production registry? — Tests understanding of prevention-versus-cleanup timing.

• Design an optimized Dockerfile for a Go service: minimal size, fast builds, and low attack surface. — Tests understanding of multi-stage builds, distroless bases, and layer ordering.