Resource Management

TL;DR

• Requests are scheduler reservations; limits are runtime hard ceilings
• CPU is compressible (throttled at limit); memory is not (OOMKilled at limit)
• QoS classes (Guaranteed > Burstable > BestEffort) determine eviction order under memory pressure
• Guaranteed requires requests == limits for all containers (both CPU and memory)
• ResourceQuota caps total namespace consumption; LimitRange provides per-container defaults and constraints
• A ResourceQuota in a namespace forces all pods to specify requests/limits or be rejected

Core Concepts

Requests and Limits

Requests and limits are distinct mechanisms that drive both scheduling decisions and runtime enforcement:

Requests represent the amount of resources the scheduler reserves for a pod. The sum of all container requests on a node cannot exceed that node’s allocatable capacity. Requests are a scheduling-time guarantee, not a runtime cap.

Limits are hard ceilings enforced by the kubelet and container runtime at execution time. Behavior differs by resource type:

• CPU is compressible: exceeding the limit triggers throttling via CFS quota; the container slows down but continues running
• Memory is not compressible: exceeding the limit triggers an immediate OOMKill (SIGKILL) with no grace period

resources:
  requests:
    cpu: "250m"      # 0.25 vCPU reserved for scheduling
    memory: "256Mi"  # 256MiB reserved for scheduling
  limits:
    cpu: "500m"      # Throttled beyond this
    memory: "512Mi"  # OOMKilled beyond this

Quality of Service (QoS) Classes

Kubernetes automatically assigns a QoS class to each pod based on its request and limit configuration. QoS class determines the eviction priority when the node experiences memory pressure.

Query a pod’s QoS class:

kubectl get pod my-pod -o jsonpath='{.status.qosClass}'

Namespace-Level Resource Controls

ResourceQuota

A ResourceQuota enforces hard limits on total resource consumption within a namespace:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-payments-quota
  namespace: payments
spec:
  hard:
    requests.cpu: "20"
    requests.memory: 40Gi
    limits.cpu: "40"
    limits.memory: 80Gi
    pods: "50"

When a ResourceQuota is present, every pod created in that namespace must specify resource requests and limits, or the API server rejects it immediately.

LimitRange

A LimitRange provides per-container defaults and min/max constraints, preventing pods without explicit resource specifications from silently becoming BestEffort:

apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
  namespace: payments
spec:
  limits:
  - default:
      cpu: 500m
      memory: 512Mi
    defaultRequest:
      cpu: 250m
      memory: 256Mi
    type: Container

Pairing ResourceQuota with LimitRange is a best practice: the quota enforces consumption limits while the LimitRange provides sane defaults and prevents resource-unspecified pods.

Comparison Table

Command Reference

Inspecting Resource Allocation and Usage

# Check allocated resources and remaining capacity per node
kubectl describe node worker-1 | grep -A10 "Allocated resources"

# View live resource usage per pod (requires metrics-server)
kubectl top pods -n payments
kubectl top nodes

# Check QoS class of a pod
kubectl get pod my-pod -o jsonpath='{.status.qosClass}'

# Query all pods with a specific QoS class
kubectl get pods --all-namespaces -o custom-columns=NAME:.metadata.name,QOS:.status.qosClass

Troubleshooting Resource Constraints

# Find why a pod is Pending (typically insufficient resources)
kubectl describe pod my-pod | grep -A5 Events
# Look for: "Insufficient cpu" or "Insufficient memory"

# Check ResourceQuota consumption in a namespace
kubectl describe resourcequota team-payments-quota -n payments

# View LimitRange constraints in a namespace
kubectl describe limitrange default-limits -n payments

# Find pods without resource requests/limits (BestEffort)
kubectl get pods --all-namespaces -o custom-columns=NAME:.metadata.name,QOS:.status.qosClass | grep BestEffort

Setting Resource Requests and Limits

# Complete example with requests and limits
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: app
        image: myapp:1.0
        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"
          limits:
            cpu: "500m"
            memory: "512Mi"

Common Pitfalls

Pitfall — No Requests/Limits = BestEffort = First to Evict A pod with no resource specification schedules onto any node with free capacity, but becomes the first casualty under memory pressure.

• Why: BestEffort pods have no priority reservation and are always evicted first when the node needs to reclaim memory.
• Fix: Enforce LimitRange to provide defaults, or implement ResourceQuota + LimitRange together to force explicit resource specifications.

Pitfall — Memory Spikes Cause OOMKill Despite Low Average Usage Metrics from kubectl top show memory usage well below the limit, yet the pod is OOMKilled.

• Why: kubectl top reports periodic samples (every 15-60 seconds) averaging usage over time. The cgroup OOM killer reacts to instantaneous spikes, killing the container before the next metrics sample.
• Fix: Profile actual peak memory usage, not average. Set limits with significant headroom or tune the application’s memory behavior directly (e.g., JVM heap sizing).

Pitfall — Guaranteed QoS Wastes Node Capacity Using Guaranteed QoS for all pods reserves full limit capacity, even when actual usage is far lower.

• Why: The scheduler treats requests == limits, preventing overcommitment and reducing bin-packing efficiency.
• Fix: Use Guaranteed only for critical stateful workloads (databases). Use Burstable for stateless web services to allow better resource utilization while still reserving a baseline request.

Pitfall — ResourceQuota Without LimitRange Causes Unexpected Rejections Pods created without explicit resource specs are rejected, causing confusion about “pod scheduling failures.”

• Why: ResourceQuota requires every pod to specify requests/limits; without a LimitRange providing defaults, unspecified pods fail API server validation.
• Fix: Always pair ResourceQuota with LimitRange to inject sensible defaults.

Pitfall — Misconfiguring Burstable QoS and Assuming Guaranteed Behavior Setting requests < limits for the expectation of guaranteed performance under contention.

• Why: Burstable pods are evicted second under memory pressure; they are not guaranteed to survive node resource exhaustion.
• Fix: Use Guaranteed for workloads requiring guaranteed placement; use Burstable consciously for trade-offs in utilization.

Interview Questions

Q — Explain the semantic difference between resource requests and limits. What does each control?

Q — How does Kubernetes determine a pod’s QoS class, and why does it matter for cluster reliability?

Q — Under node memory pressure, in what order are pods evicted, and why is BestEffort evicted first?

Q — What happens when a container exceeds its CPU limit vs. when it exceeds its memory limit? Why are they different?

Q — A pod continues to be OOMKilled even though kubectl top pod shows it using only 60% of its memory limit. What is the likely cause?

Q — Explain the operational relationship between ResourceQuota and LimitRange. Why should they be used together?

Q — Design a namespace resource policy (using ResourceQuota and LimitRange) for a shared multi-tenant cluster.

Q — When would you deliberately use Burstable QoS instead of Guaranteed, and what are the trade-offs?