Pod Priority & Preemption

TL;DR

• PriorityClass assigns numeric priority values to pods; scheduler schedules higher-priority pods first
• Preemption evicts lower-priority pods to make room for higher-priority pods when cluster is full
• System-critical priorities: 1 billion and above (reserved for system components)
• preemptionPolicy: Never allows a pod to get scheduling priority but prevents it from triggering evictions
• Default priority is 0 if no PriorityClass assigned and no globalDefault exists
• Preemption can violate PodDisruptionBudget (PDB) if no other option exists; involuntary disruption
• Governance critical: prevent priority inflation where every team assigns their workload “critical” priority

Core Concepts

PriorityClass Mechanism

PriorityClass is a cluster-scoped object defining named priority levels. Pods reference a PriorityClass, inheriting its priority value and preemptionPolicy.

Example: PriorityClass “high-priority” has value 1000. Pod referencing this PriorityClass gets priority 1000. Scheduler sees numeric priority; higher number = higher priority.

Preemption

When cluster has insufficient resources and a high-priority pending pod cannot fit on any node, the scheduler looks for nodes where evicting one or more low-priority pods would create room. It evicts those pods and schedules the high-priority pod.

Evicted pods are gracefully terminated (10-second grace period by default) unless preemptionPolicy prevents it.

preemptionPolicy Options

PreemptLowerPriority (default): Pod can trigger eviction of lower-priority pods.

Never: Pod gets scheduling priority but never triggers evictions. Other higher-priority pods can still preempt it.

System Component Reservation

Kubernetes system components (kube-apiserver, etcd, kubelet, kube-proxy, etc.) use priorities ≥ 1 billion. User workload priorities should stay < 1 billion to avoid conflicts.

globalDefault

If a PriorityClass has globalDefault: true, all pods without an explicit PriorityClass inherit its priority. Useful for setting baseline priority.

Priority vs. QoS Class

Priority is explicit (PriorityClass), controls preemption and eviction order.

QoS class (Guaranteed, Burstable, BestEffort) is implicit, based on resource requests/limits, controls eviction under resource pressure.

Both work together: high-priority BestEffort pod gets scheduled first, but under memory pressure, it’s evicted before low-priority Guaranteed pod.

Comparison Table

Command Reference

Create PriorityClass

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: critical-priority
value: 1000
globalDefault: false
preemptionPolicy: PreemptLowerPriority
description: "For mission-critical workloads only"

Pod Using PriorityClass

apiVersion: v1
kind: Pod
metadata:
  name: critical-app
spec:
  priorityClassName: critical-priority
  containers:
  - name: app
    image: critical-app:v1
    resources:
      requests:
        cpu: "500m"
        memory: "512Mi"

Deployment Using PriorityClass

apiVersion: apps/v1
kind: Deployment
metadata:
  name: critical-service
spec:
  template:
    spec:
      priorityClassName: critical-priority
      containers:
      - name: service
        image: service:v1

Essential Commands

# List PriorityClasses
kubectl get priorityclass

# View specific PriorityClass
kubectl describe priorityclass critical-priority

# Check a pod's priority
kubectl get pod <pod> -o json | grep priority

# Simulate preemption (describe pending pod)
kubectl describe pod <pending-pod>
# Look for: "preemption" events in status

Common Pitfalls

Pitfall: Priority Inflation Every team assigns their workload “critical” priority; critical becomes meaningless.

• Why: Without governance, priority assignment is self-service. Teams assume their workload is most important; everyone sets “critical.”
• Fix: Establish centralized priority policy. Limit high-priority PriorityClasses to clearly-identified critical systems (payment processing, auth, etc.). Audit and enforce via RBAC (restrict who can assign high priorities) or admission controllers.

Pitfall: Preemption Violates Expected Uptime (PodDisruptionBudget) Critical pod is preempted despite having a PodDisruptionBudget (PDB) set to minAvailable: 1.

• Why: Preemption is involuntary—if no alternative exists (no other pods to evict), even PDB is violated to make room for very high-priority pod.
• Fix: Set preemptionPolicy: Never on your PriorityClass if zero tolerance for involuntary preemption. Accept that preemption can override PDB only if no other option exists. Monitor for preemption events and adjust priorities/PDBs to be more explicit.

Pitfall: Pods With No PriorityClass Get Zero Priority New pod without explicit PriorityClass never schedules; all resources busy.

• Why: Default priority is 0. If cluster has other pods with positive priorities, the zero-priority pod is last in line and may starve. No globalDefault set means no fallback priority.
• Fix: Set a globalDefault PriorityClass: globalDefault: true to automatically assign baseline priority to all pods. Ensure all critical workloads have explicitly higher priority.

Pitfall: preemptionPolicy: Never Pod Also Never Scheduled Pod with preemptionPolicy: Never remains Pending indefinitely despite node capacity.

• Why: Misunderstanding: preemptionPolicy: Never means the pod won’t trigger preemption. It doesn’t guarantee the pod will be scheduled without other pods being evicted. Scheduler still respects node capacity; if a lower-priority pod holds the only space, it won’t be evicted.
• Fix: Ensure overall cluster capacity. If preemption-safe workloads are critical, also ensure sufficient baseline resources. Or accept that truly preemption-proof workloads require either overprovisioned capacity or explicit eviction targets.

Pitfall: Preemption With PDB Conflict Causes Thrashing Pod A (high priority) preempts pod B; B restarts, preempts C; cascade of evictions.

• Why: Without careful PDB design, high-priority pod preemptions can cascade, causing multiple restarts and downtime.
• Fix: Design PriorityClasses conservatively. Reserve high priority for truly critical paths. Use preemptionPolicy: Never for workloads that should not cause cascading disruptions. Plan PodDisruptionBudgets to absorb preemption without excessive churn.

Interview Questions

Q — Explain how preemption works. When would the scheduler preempt a running pod? Preemption occurs when cluster is out of capacity and a high-priority pod cannot fit anywhere. Scheduler looks for nodes where evicting one or more lower-priority pods would free enough resources for the high-priority pod. If found, it evicts those pods (graceful termination, 10-second grace period) and schedules the high-priority pod. Preemption is automatic and involuntary; it’s a last-resort scheduling strategy when higher-priority workloads need resources.

Q — Why is governance of PriorityClass assignment critical, and how should it be enforced? Without governance, every team will assign their workload “critical” priority—priority inflation. If everyone is critical, prioritization becomes meaningless. Enforce governance via RBAC (restrict who can create PriorityClasses or reference high-priority classes) or admission controllers (webhook prevents pods from using unauthorized priorities). Define clear criteria for each priority level (payment processing = critical, internal tools = low) and audit assignments.

Q — What is the difference between priority and QoS class? How do they interact? Priority is explicit (PriorityClass), controls scheduling order and preemption; higher priority pods schedule first and can preempt lower-priority pods. QoS class (Guaranteed/Burstable/BestEffort) is implicit (based on resource requests/limits), controls eviction order under resource pressure; Guaranteed pods are evicted last. They interact: under resource pressure, high-priority BestEffort pod is scheduled first, but evicted before low-priority Guaranteed pod.

Q — Can preemption violate a PodDisruptionBudget, and when? Yes. PodDisruptionBudget is best-effort for voluntary disruptions (maintenance, drains). Preemption is involuntary; if no other option exists (no other pods can be evicted, or all have PDBs preventing eviction), preemption can violate a PDB to make room for a very high-priority pod. This is intentional—critical system stability (e.g., API server scaling up) takes precedence over disruption budgets. If zero tolerance for involuntary preemption, set preemptionPolicy: Never on your PriorityClass.