DaemonSets, Jobs & CronJobs: Node-Level and Batch Workloads

TL;DR

• DaemonSet: guarantees exactly one Pod copy on every (or a filtered subset of) node — used for log shippers, monitoring agents, CNI plugins, kube-proxy.
• Job: runs Pods to completion for a fixed amount of work, then stops — does not restart Pods that exit 0.
• CronJob: creates Jobs on a repeating cron schedule; concurrencyPolicy controls overlap behavior.
• Job restartPolicy must be Never or OnFailure — Always is rejected by the API server.
• Biggest gotcha: completed Job/CronJob pods are not garbage collected by default and silently accumulate in etcd unless ttlSecondsAfterFinished and history limits are set.

Core Concepts

DaemonSets: One Pod Per Node

A DaemonSet ensures that a copy of a Pod runs on every (or a filtered subset of) node in the cluster. As nodes are added, the DaemonSet controller automatically schedules a Pod onto them; as nodes are removed, those Pods are garbage collected. DaemonSet pods bypass the scheduler’s normal node-selection logic but still respect taints, node selectors, and resource fit — the DaemonSet controller itself decides placement, one Pod per matching node.

Typical use cases:

• Log shippers (Fluentd, Filebeat) collecting node-level logs.
• Node monitoring agents (Node Exporter, Datadog agent).
• CNI plugins and kube-proxy itself (deployed as a DaemonSet in most distributions).
• Storage daemons (Ceph, GlusterFS).

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: node-exporter
  template:
    metadata:
      labels:
        app: node-exporter
    spec:
      tolerations:
      - key: node-role.kubernetes.io/control-plane
        effect: NoSchedule
      containers:
      - name: node-exporter
        image: prom/node-exporter:v1.7.0
        ports:
        - containerPort: 9100
          hostPort: 9100
        resources:
          limits:
            memory: 128Mi
          requests:
            cpu: 100m
            memory: 64Mi

The explicit toleration for the control-plane taint is required because master nodes are tainted NoSchedule by default, so a monitoring DaemonSet that must also run there needs to tolerate it.

Jobs: Run-to-Completion Workloads

A Job creates one or more Pods and ensures a specified number of them terminate successfully. Unlike a Deployment, a Job does not restart Pods that exit with status 0 — it considers the work done.

Key fields:

• completions: total number of successful pod completions required (default 1).
• parallelism: how many pods can run concurrently.
• backoffLimit: retries before marking the Job failed (default 6).
• activeDeadlineSeconds: hard wall-clock timeout for the whole Job.
• ttlSecondsAfterFinished: auto-cleanup of finished Job objects.

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migration
spec:
  completions: 1
  parallelism: 1
  backoffLimit: 3
  activeDeadlineSeconds: 600
  ttlSecondsAfterFinished: 3600
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: migrate
        image: myorg/migrator:1.4.0
        command: ["./migrate", "up"]

restartPolicy inside a Job’s Pod template must be Never or OnFailure — Always (the Deployment default) is rejected by the API server.

Parallel work-queue pattern: setting completions: 20 and parallelism: 5 processes 20 total units of work with 5 pods running at any time.

CronJobs: Scheduled Jobs

A CronJob creates Jobs on a repeating schedule using standard cron syntax (* * * * * = minute, hour, day-of-month, month, day-of-week).

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-backup
spec:
  schedule: "0 2 * * *"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 1
  startingDeadlineSeconds: 200
  jobTemplate:
    spec:
      backoffLimit: 2
      template:
        spec:
          restartPolicy: OnFailure
          containers:
          - name: backup
            image: myorg/backup-tool:2.1
            command: ["/bin/sh", "-c", "backup.sh --target=s3"]

concurrencyPolicy options:

• Allow (default): overlapping runs are permitted.
• Forbid: skips a new run if the previous one hasn’t finished.
• Replace: cancels the currently running Job and starts the new one.

A CronJob running twice for the same schedule slot is almost always caused by concurrencyPolicy: Allow combined with a Job that runs longer than the schedule interval, or by the CronJob controller catching up on missed schedules after a control-plane outage (bounded by startingDeadlineSeconds). Setting concurrencyPolicy: Forbid for anything that must not overlap, and an explicit startingDeadlineSeconds, ensures missed runs beyond that window are skipped rather than piling up.

Job vs CronJob vs DaemonSet vs Deployment

Aspect	Deployment	DaemonSet	Job	CronJob
Pod count target	N replicas, user-defined	Exactly 1 per eligible node	completions successful runs	1 Job per schedule tick
Pod restart on exit 0	Restarted (long-running)	Restarted (long-running)	Not restarted — considered done	Not restarted per-run
Scheduler involvement	Normal scheduling	Bypasses node-selection; still respects taints/resources	Normal scheduling	Normal scheduling
Scaling model	kubectl scale / HPA	Auto-scales with node count	parallelism controls concurrency	Each tick creates a new Job
Typical use	Stateless services	Node agents (logging, monitoring, CNI)	One-off / batch tasks (migrations)	Recurring batch tasks (backups)
Cleanup concern	N/A	N/A	ttlSecondsAfterFinished for finished pods	successfulJobsHistoryLimit / failedJobsHistoryLimit

Command / Configuration Reference

# Create an ad-hoc Job imperatively
kubectl create job manual-run --image=busybox -- /bin/sh -c "echo hello"

# Trigger a CronJob's Job immediately, outside its schedule
kubectl create job --from=cronjob/nightly-backup manual-trigger-01

# Watch DaemonSet rollout status across all nodes
kubectl rollout status daemonset/node-exporter -n monitoring

# Suspend a CronJob without deleting it
kubectl patch cronjob nightly-backup -p '{"spec":{"suspend":true}}'

# Inspect a Job's pods and logs to diagnose failures
kubectl get pods --selector=job-name=db-migration
kubectl logs job/db-migration --all-containers

Common Pitfalls

• Pitfall: Completed Job pods remain in Completed state indefinitely and accumulate in the cluster. Why: Job pods are not garbage collected by default unless ttlSecondsAfterFinished is set. Fix: Always set ttlSecondsAfterFinished on Jobs and sane successfulJobsHistoryLimit / failedJobsHistoryLimit values on CronJobs to keep etcd and kubectl get pods performant.
• Pitfall: A DaemonSet Pod is stuck Pending on one specific node. Why: The node carries a taint the DaemonSet’s Pod spec doesn’t tolerate, or the node lacks sufficient allocatable CPU/memory to satisfy the Pod’s requests. Fix: Check kubectl describe node | grep Taints and add matching tolerations, or free up node resources.
• Pitfall: A Job’s Pod template is written with restartPolicy: Always. Why: Always is the Deployment-style default but is invalid for Job pods since it conflicts with the run-to-completion model. Fix: Set restartPolicy: Never or OnFailure explicitly.
• Pitfall: A CronJob appears to run its Job twice for one schedule slot. Why: concurrencyPolicy: Allow (the default) permits overlapping runs when the previous Job outlives the schedule interval, or the controller is catching up on missed runs after downtime. Fix: Set concurrencyPolicy: Forbid and a bounded startingDeadlineSeconds.

Interview Questions

• A DaemonSet Pod is stuck Pending on one specific node — what do you check first? — tests understanding that DaemonSet pods still respect taints and resource fit despite bypassing scheduler node-selection.
• Why does a Job reject restartPolicy: Always in its Pod template? — tests understanding of the run-to-completion Pod lifecycle model versus long-running services.
• What is the difference between backoffLimit and activeDeadlineSeconds on a Job? — tests precision on retry-count-based vs wall-clock-based failure limits.
• How would you design a CronJob so that a slow-running backup never overlaps with the next scheduled run? — tests knowledge of concurrencyPolicy and its trade-offs.
• Why do completed Job pods need explicit cleanup configuration? — tests awareness of ttlSecondsAfterFinished and cluster-scale operational hygiene.

A DaemonSet ensures that a copy of a Pod runs on every (or a filtered subset of) node in the cluster. As nodes are added, the DaemonSet controller automatically schedules a Pod onto them; as nodes are removed, those Pods are garbage collected.

Typical use cases:

• Log shippers (Fluentd, Filebeat) collecting node-level logs.
• Node monitoring agents (Node Exporter, Datadog agent).
• CNI plugins and kube-proxy itself (deployed as a DaemonSet in most distributions).
• Storage daemons (Ceph, GlusterFS).

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: node-exporter
  template:
    metadata:
      labels:
        app: node-exporter
    spec:
      tolerations:
      - key: node-role.kubernetes.io/control-plane
        effect: NoSchedule
      containers:
      - name: node-exporter
        image: prom/node-exporter:v1.7.0
        ports:
        - containerPort: 9100
          hostPort: 9100
        resources:
          limits:
            memory: 128Mi
          requests:
            cpu: 100m
            memory: 64Mi

Note the explicit toleration for the control-plane taint — by default, master nodes are tainted NoSchedule, so a monitoring DaemonSet that must also run there needs to tolerate it.

2. Jobs: Run-to-Completion Workloads

A Job creates one or more Pods and ensures a specified number of them terminate successfully. Unlike a Deployment, a Job does not restart Pods that exit with status 0 — it considers the work done.

Key fields:

• completions: total number of successful pod completions required (default 1).
• parallelism: how many pods can run concurrently.
• backoffLimit: retries before marking the Job failed (default 6).
• activeDeadlineSeconds: hard wall-clock timeout for the whole Job.
• ttlSecondsAfterFinished: auto-cleanup of finished Job objects.

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migration
spec:
  completions: 1
  parallelism: 1
  backoffLimit: 3
  activeDeadlineSeconds: 600
  ttlSecondsAfterFinished: 3600
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: migrate
        image: myorg/migrator:1.4.0
        command: ["./migrate", "up"]

restartPolicy inside a Job’s Pod template must be Never or OnFailure — Always (the Deployment default) is rejected by the API server.

Parallel work-queue pattern: set completions: 20 and parallelism: 5 to process 20 total units of work with 5 pods running at any time.

3. CronJobs: Scheduled Jobs

A CronJob creates Jobs on a repeating schedule using standard cron syntax (* * * * * = minute, hour, day-of-month, month, day-of-week).

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-backup
spec:
  schedule: "0 2 * * *"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 1
  startingDeadlineSeconds: 200
  jobTemplate:
    spec:
      backoffLimit: 2
      template:
        spec:
          restartPolicy: OnFailure
          containers:
          - name: backup
            image: myorg/backup-tool:2.1
            command: ["/bin/sh", "-c", "backup.sh --target=s3"]

concurrencyPolicy options:

• Allow (default): overlapping runs are permitted.
• Forbid: skips a new run if the previous one hasn’t finished.
• Replace: cancels the currently running Job and starts the new one.

4. Hands-on Command Cheat Sheet

Create an ad-hoc Job imperatively (fast for exam scenarios):

kubectl create job manual-run --image=busybox -- /bin/sh -c "echo hello"

Trigger a CronJob’s Job immediately, outside its schedule (great for testing):

kubectl create job --from=cronjob/nightly-backup manual-trigger-01

Watch DaemonSet rollout status across all nodes:

kubectl rollout status daemonset/node-exporter -n monitoring

Suspend a CronJob without deleting it:

kubectl patch cronjob nightly-backup -p '{"spec":{"suspend":true}}'

Check why a Job’s pods keep failing:

kubectl get pods --selector=job-name=db-migration
kubectl logs job/db-migration --all-containers

5. Architectural Interview Q&A

Q: A DaemonSet Pod is stuck Pending on one specific node. What do you check first? A: Check for taints on that node (kubectl describe node | grep Taints) that the DaemonSet’s Pod spec doesn’t tolerate, and check for resource pressure (the node may not have enough allocatable CPU/memory to satisfy the Pod’s requests). Unlike normal scheduling, DaemonSet pods bypass the scheduler’s node-selection but still respect taints, node selectors, and resource fit.

Q: Your nightly CronJob occasionally runs twice for the same schedule slot. Why, and how do you fix it? A: This is almost always concurrencyPolicy: Allow (the default) combined with a Job that runs longer than the schedule interval, or the CronJob controller catching up on missed schedules after a control-plane outage (bounded by startingDeadlineSeconds). Fix: set concurrencyPolicy: Forbid for anything that must not overlap, and set an explicit startingDeadlineSeconds so missed runs beyond that window are simply skipped rather than piling up.

6. Common Pitfalls

[!WARNING] Job Pods Are Not Garbage Collected By Default Completed Job pods remain in Completed state indefinitely unless you set ttlSecondsAfterFinished, or clean them up externally. On a cluster running thousands of CronJobs, this silently fills up etcd with terminal pod objects and slows down kubectl get pods and API server list calls cluster-wide. Always set ttlSecondsAfterFinished and sane successfulJobsHistoryLimit / failedJobsHistoryLimit values.