Taints & Tolerations

TL;DR

• Taints are node-level constraints that repel pods; tolerations are pod-level declarations that permit (but do not force) scheduling onto tainted nodes
• Three taint effects: NoSchedule (prevents new pod scheduling), PreferNoSchedule (soft preference), NoExecute (evicts non-tolerating pods)
• Toleration syntax: match by operator: Equal (exact value) or operator: Exists (any value)
• tolerationSeconds delays eviction on NoExecute taints; without it, eviction is immediate
• Control-plane nodes have a built-in node-role.kubernetes.io/control-plane:NoSchedule taint
• A toleration permits but does not guarantee placement — combine with nodeSelector or affinity to actually target a node

Core Concepts

Taints and Taint Effects

Taints are applied to nodes and repel pods. A taint consists of three components: key=value:effect.

Three effects control different behaviors:

Tolerations: Pod-Level Tolerance

Tolerations are pod-level specifications that allow (but do not mandate) scheduling onto tainted nodes. Two operator types define tolerance matching:

operator: Equal — Requires exact key, value, and effect match:

tolerations:
- key: "dedicated"
  operator: "Equal"
  value: "gpu-workloads"
  effect: "NoSchedule"

operator: Exists — Matches any value for the specified key; the value field is ignored. Useful for broad tolerance declarations:

tolerations:
- key: "dedicated"
  operator: "Exists"
  effect: "NoSchedule"

Toleration vs. Scheduling Guarantee

A critical distinction: tolerations only permit scheduling onto tainted nodes; they do not force placement there. The scheduler remains free to place the pod on any untainted node that satisfies resource requirements. To guarantee placement on a specific tainted node, pair the toleration with a nodeSelector or nodeAffinity.

Graceful Eviction with tolerationSeconds

The tolerationSeconds field applies only to NoExecute taints and specifies how long a pod remains on a node after the taint is applied before eviction:

tolerations:
- key: "node.kubernetes.io/not-ready"
  operator: "Exists"
  effect: "NoExecute"
  tolerationSeconds: 300

Without tolerationSeconds, eviction occurs immediately when a NoExecute taint is applied.

Built-in Taints

Kubernetes automatically applies several taints:

• node-role.kubernetes.io/control-plane:NoSchedule — Prevents regular workloads from scheduling on control-plane nodes
• node.kubernetes.io/not-ready:NoExecute — Applied when a node’s Ready condition becomes false
• node.kubernetes.io/unreachable:NoExecute — Applied when the node controller cannot reach a node
• node.kubernetes.io/memory-pressure, node.kubernetes.io/disk-pressure — Applied under resource pressure conditions

Command Reference

Applying and Removing Taints

# Apply a taint to a node
kubectl taint nodes worker-3 dedicated=gpu-workloads:NoSchedule

# Apply a taint with NoExecute effect (evicts non-tolerating pods)
kubectl taint nodes worker-3 gpu-dedicated=true:NoExecute

# Remove a taint (trailing dash syntax)
kubectl taint nodes worker-3 dedicated=gpu-workloads:NoSchedule-

# Remove all taints of a specific key
kubectl taint nodes worker-3 dedicated-

Inspecting Taints and Tolerations

# List taints on all nodes in a custom format
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

# Inspect taints on a specific node (human-readable)
kubectl describe node worker-3 | grep -A5 Taints

# Check pod tolerations
kubectl get pod my-pod -o jsonpath='{.spec.tolerations}'

# Describe a pod to see tolerations in full detail
kubectl describe pod my-pod | grep -A10 Tolerations

Common Taint Scenarios

# Untaint control-plane node to allow regular workloads (lab environments)
kubectl taint nodes controlplane node-role.kubernetes.io/control-plane:NoSchedule-

# Add a taint for a dedicated workload node pool
kubectl taint nodes worker-1 workload=analytics:NoSchedule
kubectl taint nodes worker-2 workload=analytics:NoSchedule

# List only nodes that have no taints
kubectl get nodes -o jsonpath='{.items[?(@.spec.taints==null)].metadata.name}'

Comparison Table

Common Pitfalls

Pitfall — Toleration Without Affinity Does Not Guarantee Placement Adding a toleration to a pod and expecting it to run on the tainted node is a frequent production and exam mistake.

• Why: Tolerations only permit scheduling; the scheduler can still place the pod on any untainted node that has available resources.
• Fix: Pair the toleration with a nodeSelector or nodeAffinity to explicitly target the tainted node(s).

Pitfall — Applying NoExecute Without tolerationSeconds Applying a NoExecute taint to a node with running pods evicts them immediately without a grace period.

• Why: NoExecute without tolerationSeconds triggers immediate eviction; pods have no time to shut down gracefully.
• Fix: Define tolerationSeconds in pod specs to allow graceful termination, or set taints on fresh nodes before deploying pods.

Pitfall — Forgetting the Trailing Dash for Taint Removal Attempting to remove a taint without the trailing dash fails silently or appears to modify the taint rather than remove it.

• Why: The dash syntax is required by kubectl taint; omitting it treats the command as adding or modifying a taint.
• Fix: Always include the trailing dash: kubectl taint nodes worker-1 key=value:Effect-.

Pitfall — Misunderstanding Node Drain vs. NoExecute Taint Applying a NoExecute taint is not equivalent to gracefully draining a node.

• Why: NoExecute evicts pods immediately (or after tolerationSeconds); kubectl drain respects grace periods and is safer for maintenance.
• Fix: Use kubectl drain for scheduled maintenance; use NoExecute taints only when immediate workload removal is acceptable.

Interview Questions

Q — Explain the semantic difference between NoSchedule and NoExecute taint effects and when each is appropriate.

Q — A pod has a toleration but still schedules on an untainted node. Why is this expected behavior, and how would you force it to use a specific tainted node?

Q — You apply a NoExecute taint to a node with running pods. What happens to pods with and without matching tolerations? Does order matter?

Q — How does tolerationSeconds interact with NoExecute taints, and what is the default behavior if the field is omitted?

Q — Design a taint/toleration scheme for a shared cluster where team-A owns a dedicated node pool but cluster-wide logging must still run on those nodes.

Q — What is the purpose of the built-in node-role.kubernetes.io/control-plane:NoSchedule taint, and why is it applied automatically?

Q — How does the operator: Exists toleration differ from operator: Equal, and when would you use each in practice?