Health Probes

TL;DR

• Liveness probe detects if a container is deadlocked or hung; failure triggers restart
• Readiness probe determines if a container should receive traffic; failure removes pod from Service endpoints without restart
• Startup probe provides a generous grace period for slow-booting containers before liveness/readiness checks begin
• Four probe mechanisms: httpGet, tcpSocket, exec, grpc
• Never use liveness probes to check downstream dependencies (DB, API); use readiness instead
• startupProbe decouples slow boot tolerance from fast steady-state failure detection

Core Concepts

The Three Probe Types

Kubernetes relies on three distinct probe types, each answering a different health question:

Liveness Probe — Answers “Is this container still functioning, or is it deadlocked/hung?” Failure causes the kubelet to kill and restart the container. Liveness probes should check only the health of the application process itself, not external dependencies.

Readiness Probe — Answers “Is this container ready to receive traffic right now?” Failure removes the pod from Service endpoints (traffic stops flowing to it) but does not restart the container. Readiness probes are appropriate for checking dependency connectivity, temporary resource unavailability, or application initialization completion.

Startup Probe — Answers “Has this container finished its (possibly slow) initial boot?” Failure causes restart; liveness and readiness checks are suppressed until startup succeeds. Provides a generous grace period for slow cold-starts (JVM warmup, large cache preload) without weakening steady-state liveness responsiveness.

Probe Mechanisms

Four mechanisms execute health checks:

httpGet — Sends an HTTP GET request to the specified path and port. Success requires HTTP status code 200-399. Best for HTTP/HTTPS services.

tcpSocket — Attempts to open a TCP connection to the specified port. Success requires the port to accept the connection (no HTTP semantics). Suitable for non-HTTP services like databases or raw TCP services.

exec — Runs a command inside the container. Success requires exit code 0. Flexible but can be slow; avoid expensive operations.

grpc — Uses the native gRPC health checking protocol (grpc.health.v1.Health). Available as a stable field in newer Kubernetes versions; no longer requires shell wrappers.

Why Readiness ≠ Liveness

A critical operational principle: never use liveness probes to check downstream dependencies (databases, caches, APIs). When a dependency fails, a liveness probe checking that dependency fails on all replicas simultaneously. The kubelet then restarts all replicas in a restart loop — an action that does nothing to fix the broken dependency but adds cascading failure and cold-start latency during an already-stressed incident.

Readiness probes are the correct place for dependency checks. When a pod cannot reach its database, the readiness probe fails, removing the pod from Service endpoints and stopping traffic flow. Once the dependency recovers, the pod automatically recovers and resumes receiving traffic with zero restart overhead.

startupProbe vs. initialDelaySeconds

initialDelaySeconds is a static, fixed delay applied before any liveness or readiness checks begin. It requires choosing one number for all cases, which is either too short for slow cold-starts or wasteful on every fast restart.

startupProbe decouples boot-time tolerance from steady-state responsiveness. It allows a long, generous grace period during startup (e.g., failureThreshold: 30 with periodSeconds: 10 = 300 seconds), while liveness remains tight and responsive once the application is running (e.g., failureThreshold: 3). This provides both patience during startup and fast failure detection afterward.

Configuration Reference

Complete Multi-Probe Example

containers:
- name: web
  image: myorg/web:2.0
  startupProbe:
    httpGet:
      path: /healthz
      port: 8080
    failureThreshold: 30
    periodSeconds: 10       # Allows up to 300 seconds for boot
  livenessProbe:
    httpGet:
      path: /healthz
      port: 8080
    initialDelaySeconds: 0  # startupProbe covers boot time
    periodSeconds: 10
    failureThreshold: 3
    timeoutSeconds: 2
  readinessProbe:
    httpGet:
      path: /ready
      port: 8080
    periodSeconds: 5
    failureThreshold: 2
    timeoutSeconds: 1

Probe Mechanism Examples

# HTTP probe
livenessProbe:
  httpGet:
    path: /health
    port: 8080
    scheme: HTTP
  periodSeconds: 10

# TCP probe for database connectivity check (use in readiness, not liveness)
readinessProbe:
  tcpSocket:
    port: 5432
  periodSeconds: 5

# Exec probe with a command
livenessProbe:
  exec:
    command: ["cat", "/tmp/healthy"]
  periodSeconds: 10

# gRPC probe
livenessProbe:
  grpc:
    port: 5000
    service: myapp.HealthCheck
  periodSeconds: 10

Command Reference

Inspecting Probe Configuration

# View liveness probe configuration for a pod
kubectl get pod my-pod -o jsonpath='{.spec.containers[0].livenessProbe}'

# View all probe types for a container
kubectl get pod my-pod -o jsonpath='{.spec.containers[0].{livenessProbe,readinessProbe,startupProbe}}'

# Describe pod and see full probe details
kubectl describe pod my-pod

Monitoring Probe Activity

# Watch probe failures in pod events
kubectl describe pod my-pod | grep -A10 Events
# Look for: "Liveness probe failed" / "Readiness probe failed" / "Startup probe failed"

# Get latest probe events in JSON format
kubectl get events --field-selector involvedObject.name=my-pod -o json | \
  jq '.items[] | select(.reason | contains("Probe")) | {reason, message}'

# Check if pod is receiving traffic (readiness gates Service membership)
kubectl get endpoints my-service -o wide

Manual Probe Simulation

# Simulate an HTTP liveness probe from inside the container
kubectl exec -it my-pod -- curl -sf localhost:8080/healthz

# Simulate a readiness probe
kubectl exec -it my-pod -- curl -sf localhost:8080/ready

# Run the exec probe command manually
kubectl exec -it my-pod -- cat /tmp/healthy

# Test TCP connectivity (if using tcpSocket)
kubectl exec -it my-pod -- nc -zv localhost 5432

Comparison Table

Probe	Checks	Failure Action	Typical Use	Should Check Dependencies
Liveness	Process alive, not deadlocked	Kill and restart	Detect hangs/deadlocks	No — causes cascading failure
Readiness	Ready to receive traffic	Remove from endpoints	Dependency availability	Yes — graceful traffic drain
Startup	Boot completed	Kill and restart (suppresses other probes)	Initial boot completion	No — only app startup state

Common Pitfalls

Pitfall — Liveness Probe Checking Downstream Dependencies Liveness probes call external services (databases, APIs) to determine health, causing cascading failures when dependencies hiccup.

• Why: When the dependency fails, all replicas fail liveness simultaneously, restarting in a loop — an action that cannot fix the broken dependency and adds cold-start overhead during incidents.
• Fix: Liveness checks only the container process itself. Move dependency checks to the readiness probe, which gracefully stops traffic without restarting.

Pitfall — Pod Restarting in CrashLoopBackOff With No Visible Application Errors Application logs show no errors, but the pod keeps restarting every 30 seconds.

• Why: Liveness probe is failing silently; kubelet terminates the container via SIGKILL/SIGTERM, which may not appear in application logs.
• Fix: Check kubectl describe pod events for “Liveness probe failed” messages. Verify probe timeouts and ensure the health endpoint is fast and reliable.

Pitfall — Using initialDelaySeconds for Slow Boot Times Containers that bootstrap slowly (JVM, large cache) get a fixed 30-60 second delay, wasting time on fast restarts.

• Why: initialDelaySeconds is static; increasing it to cover slow cold-starts wastes time on every restart.
• Fix: Use startupProbe to decouple slow-boot tolerance (long failureThreshold, slow periodSeconds) from steady-state liveness sensitivity (short failureThreshold, fast periodSeconds).

Pitfall — Readiness Probe Checks Expensive Operations Readiness probes query databases or process large datasets, causing high load when running frequently.

• Why: Readiness probes run frequently (default 10 seconds, often configured to 5 or faster). Expensive checks compound the load.
• Fix: Readiness probes should be lightweight. Cache dependency availability checks or use separate lightweight endpoints for readiness/liveness.

Pitfall — Missing Readiness Probe, Traffic Sent to Initializing Pods Services route traffic to pods that haven’t completed initialization.

• Why: Without a readiness probe, pods appear ready immediately upon creation, before startup sequences complete.
• Fix: Always define a readiness probe; use startupProbe for slow initialization if needed.

Interview Questions

Q — Explain the semantic differences between liveness, readiness, and startup probes. What happens when each fails?

Q — Why should liveness probes never check downstream dependencies? What is the operational consequence?

Q — A pod is restarting every 30 seconds with no error logs. Walk through the troubleshooting steps.

Q — Design a probe strategy for a Java application with a 2-minute JVM startup time.

Q — When would you use tcpSocket over httpGet for a readiness probe?

Q — Explain why startupProbe with generous timeouts is preferable to increasing initialDelaySeconds for slow boot times.

Q — A pod is marked as “Ready” but still receiving no traffic. What could cause this? (Hint: consider Service selector matching)

Q — How would a readiness probe checking database connectivity help during a database outage, compared to a liveness probe doing the same check?