Progressive Delivery

TL;DR

• Progressive delivery automates the judgment call during canary rollouts: metric-based analysis decides whether to progress, pause, or roll back, replacing human dashboard-watching.
• Argo Rollouts uses custom Rollout resources with AnalysisTemplates; Flagger leverages a service mesh (Istio, Linkerd) or ingress for traffic shifting — different architectures, same outcome.
• AnalysisTemplate success criteria must be kept current; stale thresholds produce false positives (blocks good deploys) or false negatives (passes bad ones) as traffic baselines shift.
• Automated rollback supplements, never replaces, manual procedures; automation only catches the failure modes it’s configured to detect.
• Precise, percentage-based traffic shifting (not just replica scaling) is the technical requirement Flagger depends on; a broken mesh layer silently breaks the whole mechanism.

Core Concepts

Argo Rollouts and canary traffic management

Argo Rollouts replaces a standard Kubernetes Deployment with a Rollout custom resource that understands traffic percentages and analysis gates natively. A Rollout defines explicit canary steps:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: myapp
spec:
  replicas: 10
  strategy:
    canary:
      steps:
      - setWeight: 10         # Route 10% to canary
      - pause:
          duration: 5m
      - analysis:
            templates:
            - name: success-rate
      - setWeight: 50
      - pause:
          duration: 5m
      - analysis:
            templates:
            - name: success-rate

Each step shifts traffic percentage, pauses for human observation or automated analysis, and gates progression on analysis success. This is precise, repeatable, and auditable.

AnalysisTemplate: automated decision-making

AnalysisTemplate queries metrics (Prometheus, DataDog, etc.) to evaluate success criteria, automating the judgment call:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  metrics:
  - name: error_rate
    interval: 60s
    successCriteria: result < 5        # Max 5% error rate
    provider:
      prometheus:
        address: http://prometheus:9090
        query: rate(http_errors[5m]) / rate(http_requests[5m])

The Rollout controller queries this at each step, automatically proceeding if criteria pass, holding if uncertain, or rolling back if criteria fail. This removes the “person watching dashboard at 2am” requirement.

Flagger: mesh-based progressive delivery

Flagger uses a service mesh (Istio, Linkerd) or ingress controller for traffic shifting, rather than a custom Rollout resource. Same goal, different architecture. The mesh layer provides precise, percentage-based control: routing exactly 10% of live production traffic to the canary, independent of pod count.

Why not just scale pods? Replica count scaling only indirectly influences traffic share (via load-balancer round-robin); a canary with 1 pod gets 1 share, but if traffic isn’t evenly distributed, that doesn’t reliably mean 10%. The mesh layer provides explicit control: “send exactly 10% of traffic to this service, regardless of pod count.”

Analysis criteria discipline and stale thresholds

Analysis thresholds set once and never revisited become stale:

• Before a traffic pattern shift: threshold=3% error rate (normal baseline)
• After a traffic pattern shift: same threshold now produces false positives (blocks healthy deploys at 2.5% because the new normal is 1.5%) or false negatives (passes bad canaries because new normal is 4%)

Analysis criteria require periodic review tied to how system baselines evolve, not a “set once during setup and forget” approach.

Automated rollback vs. manual procedures

Automated rollback only catches failure modes it’s configured to detect. Novel failures (cascading failures from infrastructure changes, dependency timeouts not covered by configured metrics) or an outage of the metrics system itself during the rollout means automation can’t help. Every team needs a documented, tested manual rollback procedure for exactly these cases — treating progressive delivery as “we no longer need manual rollback” is a failure waiting to happen.

Argo Rollouts vs Flagger vs Blue-Green vs Canary

Configuration Reference

# Argo Rollouts canary with AnalysisTemplate
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: basic-metrics
spec:
  metrics:
  - name: error_rate
    interval: 1m
    count: 3                            # Run 3 times
    successCriteria: result < 1         # Pass if all < 1%
    provider:
      prometheus:
        address: http://prometheus:9090
        query: rate(errors_total[1m])

---
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: canary-app
spec:
  strategy:
    canary:
      steps:
      - setWeight: 20
      - pause: {duration: 2m}
      - analysis:
          templates:
          - name: basic-metrics
          args:
          - name: query
            value: "custom_query_here"
      - setWeight: 50
      - pause: {duration: 5m}
      - analysis:
          templates:
          - name: basic-metrics

# Manual rollback procedure (documented, tested)
# 1. kubectl patch rollout canary-app -p '{"spec":{"promotionPolicy":"Failure"}}'
# 2. kubectl argo rollouts abort canary-app

Common Pitfalls

• Pitfall: High-risk service still uses plain Deployment rolling update. Why: Progressive delivery tooling was never implemented. Fix: Implement Argo Rollouts or Flagger for customer-facing, blast-radius services; treat manual canary as the legacy antipattern.

• Pitfall: Analysis thresholds are stale; false positives block healthy deploys, false negatives pass bad ones. Why: Thresholds were set once during initial setup and never revisited as traffic patterns evolved. Fix: Periodically review thresholds against current baselines, especially after product or traffic pattern changes.

• Pitfall: Flagger’s traffic shifting stopped working; nobody noticed until a critical bug couldn’t roll back cleanly. Why: Service mesh configuration degraded undetected; Flagger was silently falling back to imprecise replica scaling. Fix: Monitor mesh health as part of progressive-delivery reliability; verify mesh traffic-shifting is actually working as expected.

• Pitfall: Team confident in automated rollback; during a real incident where the metrics system was down, nobody remembered the manual rollback steps. Why: Manual procedures weren’t maintained or tested alongside automation. Fix: Document and periodically test manual rollback procedures; automation supplements, never replaces, manual capability.

• Pitfall: Canary analysis configuration is copy-pasted from examples without understanding actual thresholds. Why: Thresholds weren’t derived from the actual system’s baselines. Fix: Baseline-tune every AnalysisTemplate; thresholds should reflect what “normal” means for this specific service.

Interview Questions

• Explain mechanically what happens during a Rollout’s AnalysisTemplate step. — Tests understanding of metric querying, success criteria evaluation, and automatic decision-making.

• Why does Flagger require a service mesh instead of just scaling pods to control traffic percentage? — Tests understanding of precise traffic-control requirements versus indirect load-balancer influence.

• Your team relies entirely on automated rollback. An incident occurs where the metrics system is down. What’s missing? — Tests understanding of automation’s limits and manual procedure necessity.

• A canary’s analysis shows 4% error rate on step one; your AnalysisTemplate threshold is 5%. It passes, but later causes a customer incident. What’s the root cause? — Tests understanding of stale thresholds and baseline-dependent criteria.

• Design a progressive delivery strategy for a critical financial service that also has a fast manual rollback option. — Tests holistic understanding of automation and fallback procedures.