Deployment Strategies

TL;DR

• A deployment strategy is a trade-off between blast radius (how much of production is exposed to a bad change), exposure time, and rollback speed.
• Rolling deployments are resource-efficient but run old and new versions simultaneously, creating a version-skew window.
• Blue-green keeps a full duplicate environment warm, making rollback an instant traffic-router flip — at the cost of 2x capacity.
• Canary limits exposure to a small percentage of traffic while evaluating; progressive delivery automates that evaluation and the rollback decision.
• The biggest gotcha across all strategies: rollback plans routinely ignore database migrations, and a non-backward-compatible migration can turn “rollback” into a second outage.

Core Concepts

Rolling Deployment

Replaces old instances with new ones incrementally, typically a few pods at a time. Resource-efficient — no need for double capacity — but introduces a specific risk during the rollout window: old and new versions serve traffic simultaneously. If the new version changes an API contract, a database expectation, or a shared cache format in a way the old version can’t handle (or vice versa), that version-skew window can produce inconsistent behavior. This class of bug is notoriously hard to reproduce after the fact because it disappears once the rollout completes fully in either direction.

Blue-Green Deployment

Maintains two full environments — blue (current) and green (new) — with the old environment staying fully running and warm even after green takes over traffic. This is the mechanism that makes rollback near-instant: rollback is just flipping the router/load balancer back to blue, which never stopped running. By contrast, rolling back a rolling deployment means re-deploying the old version pod by pod, taking as long as the original rollout did — precisely when speed matters most. The real cost of blue-green is running double capacity, at least during the transition window.

Canary Deployment

Routes a small percentage of real traffic (5%, for example) to the new version and evaluates it before progressing further. The blast-radius protection over a rolling deployment is specific: a rolling deployment, even done gradually, still ends with 100% of users on the new version regardless of whether it’s actually healthy, unless something explicitly halts and reverses the rollout. A canary limits exposure to that small percentage while it’s being evaluated.

Progressive Delivery

Turns canary deployment from a manual practice into an automated one. Tools like Argo Rollouts or Flagger automate the metric analysis (comparing the canary’s error rate/latency against a baseline) and the rollback decision itself. A manual canary still requires a human watching a dashboard and deciding when to progress or abort; progressive delivery catches and reverts a bad deploy in minutes with no one needing to be actively watching at that exact moment — significant for deploys outside business hours or under a fast release cadence.

Rollbacks and Database Migrations

Every strategy above assumes rollback works, and the part most rollback plans fail to account for is the database migration. A rollback plan defined only as “redeploy the previous Git commit” ignores that the application version being rolled back to may be incompatible with the database schema state the new version’s migration already applied. If the migration wasn’t backward-compatible — a dropped column, a renamed field the old code doesn’t expect — rolling back the application code alone leaves the database in a state the old code can’t handle, and “rollback” silently becomes “rollback plus a new, different outage.”

The fix: design migrations to be backward-compatible with the previous application version, using additive changes and the expand-then-contract pattern:

1. Add the new column (expand) — old code still works, ignoring it.
2. Deploy code that writes to both old and new columns.
3. Migrate/backfill existing data.
4. Deploy code that reads only the new column.
5. Drop the old column in a later, separate release (contract).

This keeps rollback genuinely available at every intermediate step, rather than a single migration burning the bridge back.

Blue-Green vs Canary vs Rolling Deployments

Recreate vs Zero-Downtime Strategies

Recreate terminates all old instances before starting new ones — the simplest strategy, but it guarantees a downtime window and is only acceptable when brief unavailability is tolerable (e.g., some batch or internal systems). Rolling, blue-green, and canary are all zero-downtime strategies by design: at least one version is always serving traffic. The trade-off recreate avoids is version skew entirely (only one version is ever live), at the cost of availability during the cutover.

Command / Configuration Reference

# Kubernetes rolling update configuration (Deployment spec)
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%   # max pods that can be unavailable during rollout
      maxSurge: 25%         # max pods created above desired count during rollout

kubectl rollout status deployment/myapp        # watch rollout progress
kubectl rollout undo deployment/myapp          # roll back to previous revision
kubectl rollout undo deployment/myapp --to-revision=3   # roll back to a specific revision
kubectl rollout history deployment/myapp       # list revision history

# Argo Rollouts canary step definition
spec:
  strategy:
    canary:
      steps:
        - setWeight: 5      # send 5% of traffic to canary
        - pause: {duration: 5m}
        - setWeight: 25
        - pause: {duration: 10m}
        - setWeight: 100

Common Pitfalls

• Pitfall: An intermittent, hard-to-reproduce bug appears during a rolling deployment and “disappears” once the rollout finishes. Why: The bug only manifested during the version-skew window when old and new versions were serving traffic together. Fix: Explicitly test API/data contract compatibility between consecutive versions before rollout, not just each version in isolation.
• Pitfall: A rollback re-deploys old application code while the database migration from the new version stays applied, causing a second outage. Why: The rollback plan only covered application code, not schema state. Fix: Treat database migrations as part of the rollback plan; use backward-compatible, expand-then-contract migrations.
• Pitfall: Blue-green’s instant-rollback capability silently disappears. Why: The idle “blue” environment was scaled down to save cost, so it’s no longer warm and ready. Fix: Keep the idle environment running at production-equivalent capacity, or accept and document the reduced rollback guarantee.
• Pitfall: A “small change” canary skips manual evaluation and causes an outage. Why: Canary evaluation was treated as optional overhead rather than a required safety gate. Fix: Apply the same evaluation gate to every canary regardless of perceived change size.
• Pitfall: A non-backward-compatible migration ships without anyone flagging the loss of rollback. Why: Migration review doesn’t check compatibility with the currently-deployed application version. Fix: Require an explicit backward-compatibility check (or documented exception) for every schema migration.

Interview Questions

• Explain exactly why blue-green rollback is faster than rolling-deployment rollback, mechanically. — Tests whether the “old environment stays warm” mechanism is understood, not just the terminology.
• A deployment ships a database migration alongside new application code. Design a rollback-safe approach. — Tests whether expand-then-contract or an equivalent backward-compatible migration pattern is the instinct.
• When would you choose progressive delivery over a manually-monitored canary, and what does it actually automate? — Tests whether the human-in-the-loop gap is understood as the real thing being solved.
• Compare the capacity cost of rolling vs blue-green vs canary deployments. — Tests understanding of the resource trade-offs, not just rollback speed.
• What class of bug does a rolling deployment expose that recreate deployments don’t? — Tests understanding of version-skew risk during mixed-version windows.