Static Pods, Scheduling, Labels & Selectors

TL;DR

• Static Pods: managed directly by kubelet from local manifest files in /etc/kubernetes/manifests/, independent of API server or scheduler
• Mirror Pods: read-only reflections of static pods in the API; changes must go to the manifest file on disk, not via kubectl edit
• Manual Scheduling: set spec.nodeName to bypass scheduler and place a pod directly on a node
• Labels: arbitrary key/value pairs attached to objects for organization and filtering
• Selectors: query language for labels using equality-based (=, !=) or set-based (in, notin, exists) syntax
• Deployment Selectors: immutable after creation to prevent ambiguity about which pods the controller owns

Core Concepts

Static Pods

Static Pods are managed directly by the kubelet on a specific node, operating independently of the API server or any controller. The kubelet watches a designated manifest directory and creates or removes pods to match the files present — even when the API server is completely down. Kubernetes uses static pods to bootstrap kubeadm clusters; the control plane components (kube-apiserver, kube-scheduler, kube-controller-manager, and optionally etcd) all run as static pods on the control node during cluster initialization.

Mirror Pods are read-only reflections of static pods that appear in the API server for visibility via kubectl get pods -n kube-system. These mirror pods cannot be edited or deleted through the API; modifications require direct changes to the manifest file on the node. The kubelet continuously syncs mirror pods back to match the persistent manifest file on disk, making kubectl edits silently revert within seconds.

Manual Scheduling via spec.nodeName

Every Pod object contains a spec.nodeName field. Under normal operation, kube-scheduler assigns a value to this field. Setting spec.nodeName manually at pod creation time bypasses the scheduler entirely:

apiVersion: v1
kind: Pod
metadata:
  name: manually-scheduled
spec:
  nodeName: worker-node-2
  containers:
  - name: nginx
    image: nginx:1.25

When the scheduler is unavailable or intentionally bypassed, manually scheduling ensures pods reach their target nodes. The binding process also occurs via the Binding subresource (used internally by the scheduler) for pods already created in the Pending state.

Labels and Selectors: Metadata Organization

Labels are arbitrary key/value pairs attached to Kubernetes objects (environment: production, tier: frontend, app: payments). They serve as the primary mechanism for organizing, grouping, and selecting resources across controllers, Services, and NetworkPolicies.

Equality-based selectors (=, ==, !=) provide simple key-value matching:

• environment=production — resources where environment equals production
• environment!=production — resources where environment does not equal production

Set-based selectors (in, notin, exists) enable more complex queries:

• environment in (production, staging) — key has one of multiple values
• tier notin (frontend) — key exists but is not frontend
• app — key exists with any value
• !app — key does not exist

In Deployment, ReplicaSet, and similar objects, spec.selector.matchLabels must match or be a subset of spec.template.metadata.labels. This constraint is validated at creation and the selector becomes immutable after creation to prevent ambiguity about pod ownership.

Comparison Table

Command Reference

Locating and Managing Static Pod Manifests

# Find the static pod manifest directory on any node
cat /var/lib/kubelet/config.yaml | grep staticPodPath
# Default: /etc/kubernetes/manifests

# List all static pods running on a node (requires node access)
ls /etc/kubernetes/manifests/

Label Operations

# Add a label to a pod
kubectl label pods my-pod environment=production

# Overwrite an existing label
kubectl label pods my-pod environment=staging --overwrite

# Remove a label (using trailing dash)
kubectl label pods my-pod environment-

Selector Queries

# Equality-based selection
kubectl get pods -l environment=production
kubectl get pods -l 'environment!=production'

# Set-based selection
kubectl get pods -l 'environment in (production, staging)'
kubectl get pods -l 'tier notin (frontend)'
kubectl get pods -l 'app'              # key exists with any value
kubectl get pods -l '!app'             # key does not exist

# Display labels as columns
kubectl get pods --show-labels
kubectl get pods -L app,tier,environment

# Find the node assignment for a pod
kubectl get pod my-pod -o jsonpath='{.spec.nodeName}'

Modifying Static Pods

# INCORRECT: Using kubectl edit on a static pod (changes revert within seconds)
kubectl edit pod kube-apiserver-controlplane -n kube-system  # Will not persist

# CORRECT: Edit the manifest file on the node
sudo vim /etc/kubernetes/manifests/kube-apiserver.yaml
# kubelet detects the change and reconciles automatically

# Temporarily disable a static pod (move the manifest)
sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/
# kubelet removes the pod; restore it to recreate

Common Pitfalls

Pitfall — Editing Static Pods via kubectl edit Running kubectl edit pod kube-apiserver-controlplane -n kube-system appears successful, but changes silently revert within seconds as kubelet reconciles the mirror pod back to the manifest file on disk.

• Why: Mirror pods are read-only reflections; kubelet continuously syncs them to match the persistent manifest file.
• Fix: Edit the YAML file directly in /etc/kubernetes/manifests/ on the node. The kubelet watches this directory and applies changes automatically.

Pitfall — Pods Stay Pending When Scheduler Is Down New pods created while kube-scheduler is unavailable remain in Pending state indefinitely.

• Why: Without setting spec.nodeName or having the scheduler available to assign nodes, pods have no target.
• Fix: Either restore the scheduler or manually set spec.nodeName in the pod spec at creation time to specify the target node.

Pitfall — Changing a Deployment’s Selector After Creation Attempting to update spec.selector on an existing Deployment fails with an immutability error.

• Why: The ReplicaSet controller uses spec.selector to determine pod ownership. Changing it would either orphan running pods or incorrectly adopt unrelated pods, creating dangerous ambiguity.
• Fix: Create a new Deployment with the desired selector rather than modifying an existing one.

Pitfall — Selector Not Matching Pod Labels Setting spec.selector.matchLabels that do not match spec.template.metadata.labels causes the pod replica count to mismatch.

• Why: The controller only manages pods matching the selector; unmatched pods are ignored, causing replica counts to diverge.
• Fix: Ensure spec.selector.matchLabels is a subset of (or exactly matches) the labels in spec.template.metadata.labels.

Interview Questions

Q — On a kubeadm cluster, which component directly manages static pods, and why can they run even when the API server is down?

Q — Explain the relationship between mirror pods and static pods. Why are mirror pods read-only through the API?

Q — How does setting spec.nodeName manually differ from relying on kube-scheduler for pod placement?

Q — What is the difference between equality-based selectors and set-based selectors? Provide examples of when each is used.

Q — Why does Kubernetes make a Deployment’s spec.selector immutable after creation? What would happen if it were mutable?

Q — You edit a static pod’s manifest file in /etc/kubernetes/manifests/. How long does it typically take for the change to be reflected, and where can you verify the change was applied?

Q — A Deployment’s spec.selector.matchLabels is app: web, but spec.template.metadata.labels has app: web and version: v2. Why is this valid, and what would happen if you reversed them?