Kubernetes Architecture

TL;DR

• Kubernetes splits into a Control Plane (cluster brain: API Server, etcd, Scheduler, Controller Manager) and worker nodes (data plane: kubelet, kube-proxy, container runtime).
• The API Server is the only component that talks to etcd; every other component — including kubelet and the schedulers/controllers — only ever talks to the API Server.
• etcd is pure storage with no active logic; it is the single source of truth for cluster state and is written to via Raft consensus.
• If etcd goes down, running pods keep running (kubelet/kube-proxy hold local state), but nothing can be scheduled, scaled, or reconciled until it recovers.
• etcd is latency-sensitive: it commits synchronously to disk, so slow storage causes timeouts and leader-election churn.

Core Concepts

System Architecture

Kubernetes follows a master/worker split. Master nodes run the Control Plane, which manages desired cluster state; worker nodes run the Data Plane, which executes the actual containers.

 ┌────────────────────────────────────────────────────────┐
 │ CONTROL PLANE (Master Nodes)                           │
 │                                                        │
 │  ┌──────────────┐     ┌──────────────┐     ┌────────┐  │
 │  │ Controller   │◀───▶│  API Server  │◀───▶│ Scheduler│  │
 │  │ Manager      │     │  (kube-apiserver)  │     └────────┘  │
 │  └──────────────┘     └──────┬───────┘                 │
 │                              │                         │
 │                              ▼                         │
 │                       ┌──────────────┐                 │
 │                       │     etcd     │                 │
 │                       └──────────────┘                 │
 └──────────────────────────────┬─────────────────────────┘
                                │ HTTPS (Secure Port 6443)
                                ▼
 ┌────────────────────────────────────────────────────────┐
 │ WORKER NODES                                           │
 │                                                        │
 │  ┌───────────────────────┐   ┌───────────────────────┐ │
 │  │ Node 1                │   │ Node 2                │ │
 │  │  ┌──────────┐         │   │  ┌──────────┐         │ │
 │  │  │ kubelet  │         │   │  │ kubelet  │         │ │
 │  │  └────┬─────┘         │   │  └────┬─────┘         │ │
 │  │       ▼               │   │       ▼               │ │
 │  │  ┌──────────┐         │   │  ┌──────────┐         │ │
 │  │  │ Container│         │   │  │ Container│         │ │
 │  │  │ Runtime  │         │   │  │ Runtime  │         │ │
 │  │  └──────────┘         │   │  └──────────┘         │ │
 │  │  ┌──────────┐         │   │  ┌──────────┐         │ │
 │  │  │kube-proxy│         │   │  │kube-proxy│         │ │
 │  │  └──────────┘         │   │  └──────────┘         │ │
 │  └───────────────────────┘   └───────────────────────┘ │
 └────────────────────────────────────────────────────────┘

Control Plane Components

Control plane components can run on a single master node or be distributed across multiple masters for high availability.

API Server (kube-apiserver)

• The single entry point to the cluster: CLI tools, worker nodes, and controllers communicate only with the API Server, never directly with each other or with etcd.
• Exposes a RESTful API over HTTPS (default secure port 6443).
• Authenticates callers, authorizes requests via RBAC, validates and admits payloads, and writes accepted changes to etcd.
• Stateless — it holds no cluster state itself; all state lives in etcd.

etcd (state storage)

• A highly available, distributed key-value store holding the complete cluster state — the source of truth.
• Uses the Raft consensus algorithm for leader election and log replication across members.
• Performs storage only; no active reconciliation logic runs here.
• Only the API Server has read/write access to etcd — no other component talks to it directly.

Scheduler (kube-scheduler)

• Watches for newly created Pods with no node assigned (nodeName empty).
• Runs a Filtering phase (predicates) to discard nodes that cannot host the pod, followed by a Scoring phase (priorities) to rank remaining candidates.
• Binds the pod to the highest-scoring node by writing nodeName back through the API Server.

Controller Manager (kube-controller-manager)

• Runs the core reconciliation loops that continuously drive actual state toward desired state.
• Bundles multiple built-in controllers in a single binary, including:
  • Node Controller — monitors node health and handles outages/evictions.
  • Deployment Controller — manages rollout of Deployments into ReplicaSets.
  • EndpointSlice Controller — keeps Service-to-Pod endpoint mappings current.

Worker Node Components

kubelet

• A node-local agent running on every node; communicates directly with the API Server (the only worker-side component that does).
• Receives PodSpecs assigned to its node and reconciles them against local container state.
• Talks to the container runtime via the Container Runtime Interface (CRI) to start/stop containers.
• Runs liveness/readiness/startup probes and reports node and pod status back to the API Server.

kube-proxy

• Runs on every node and manages host-level network rules.
• Translates stable Service IPs into actual backing Pod IPs by programming rules into iptables or IPVS.

Container Runtime (CRI)

• The software that actually creates and runs containers, e.g. containerd, CRI-O.
• Kubelet invokes it through the standardized CRI gRPC interface, decoupling Kubernetes from any specific runtime implementation.

Request Flow: kubectl apply -f deployment.yaml

1. Client-side — kubectl validates the YAML structure and issues an HTTP POST to the API Server.
2. AuthN/AuthZ — the API Server verifies the client certificate/token (authentication) and checks RBAC rules (authorization).
3. etcd write — the API Server persists the Deployment object to etcd.
4. Deployment Controller — detects the new Deployment, creates a corresponding ReplicaSet object in etcd, and exits.
5. ReplicaSet Controller — detects the ReplicaSet, creates Pod specs in etcd with empty nodeName fields.
6. Scheduler — detects unscheduled Pods, evaluates nodes via filtering/scoring, and writes the chosen nodeName back to etcd.
7. Kubelet — the kubelet on the selected node detects the now-scheduled Pod, instructs the local container runtime (via CRI) to start containers, and reports status back to the API Server.

Control Plane vs Worker Node Components

Aspect	Control Plane	Worker Node
Talks to etcd	Yes — only kube-apiserver	No — never directly
Primary role	Cluster-wide decisions and desired-state storage	Running and reporting on actual containers
Key components	API Server, etcd, Scheduler, Controller Manager	kubelet, kube-proxy, container runtime
Survives etcd outage	No — scheduling, scaling, reconciliation halt	Yes — existing pods keep running via local state
Communication pattern	Components talk to each other via the API Server	kubelet/kube-proxy talk only to the API Server

Command / Configuration Reference

# Check health of self-hosted control plane components (static pods in kube-system)
kubectl get pods -n kube-system -l tier=control-plane

# Tail kubelet logs on a worker node (systemd-managed)
journalctl -u kubelet -n 100 --no-pager

# List worker nodes with extended info (internal IP, OS image, kubelet version)
kubectl get nodes -o wide

# Confirm which component owns a given object by inspecting events
kubectl get events -n kube-system --sort-by='.lastTimestamp'

Common Pitfalls

• Pitfall: Assuming an etcd outage takes down running application pods immediately. Why: kubelet and kube-proxy cache local state (running container list, iptables rules) independent of the control plane. Fix: Understand that only cluster-level operations (deploy, scale, evict) stop working — treat etcd recovery as urgent but not as an application-outage emergency.
• Pitfall: Running etcd on slow disks (e.g. unprovisioned-IOPS network volumes). Why: etcd commits every write synchronously to disk before acknowledging; high write latency causes request timeouts and repeated leader elections. Fix: Back etcd with fast, low-latency SSD storage and monitor etcd disk fsync/commit latency metrics.
• Pitfall: Expecting components to talk to etcd directly for a “fast path.” Why: Only the API Server is authorized and wired to access etcd; bypassing it breaks the authentication/authorization/admission pipeline. Fix: Always go through the API Server, even for tooling and controllers.
• Pitfall: Confusing “node NotReady” with “control plane down.” Why: A node can go NotReady due to kubelet, network, or CNI issues while the control plane is fully healthy. Fix: Check kubectl get nodes and kubelet logs on the affected node before assuming a control plane failure.

Interview Questions

• If etcd crashes completely, does the application running on worker nodes go down? — tests understanding of the control plane/data plane separation and what kubelet/kube-proxy cache locally.
• Walk through what happens end-to-end when you run kubectl apply -f deployment.yaml. — tests whether the candidate can trace the full object flow through API Server, etcd, Deployment Controller, ReplicaSet Controller, Scheduler, and kubelet.
• Why is etcd so sensitive to disk I/O latency, and how would you provision storage for it? — tests operational awareness of etcd’s synchronous write model and Raft consensus.
• Which component is the only one with direct read/write access to etcd, and why does that matter for security? — tests understanding of the API Server as the sole gatekeeper.
• What is the difference between the Scheduler’s filtering and scoring phases? — tests depth on how pod placement decisions are actually made.

Kubernetes follows a master/worker architecture. The master nodes run the Control Plane (managing cluster state), while worker nodes run the actual Data Plane (running containers).

 ┌────────────────────────────────────────────────────────┐
 │ CONTROL PLANE (Master Nodes)                           │
 │                                                        │
 │  ┌──────────────┐     ┌──────────────┐     ┌────────┐  │
 │  │ Controller   │◀───▶│  API Server  │◀───▶│ Scheduler│  │
 │  │ Manager      │     │  (kube-apiserver)  │     └────────┘  │
 │  └──────────────┘     └──────┬───────┘                 │
 │                              │                         │
 │                              ▼                         │
 │                       ┌──────────────┐                 │
 │                       │     etcd     │                 │
 │                       └──────────────┘                 │
 └──────────────────────────────┬─────────────────────────┘
                                │ HTTPS (Secure Port 6443)
                                ▼
 ┌────────────────────────────────────────────────────────┐
 │ WORKER NODES                                           │
 │                                                        │
 │  ┌───────────────────────┐   ┌───────────────────────┐ │
 │  │ Node 1                │   │ Node 2                │ │
 │  │  ┌──────────┐         │   │  ┌──────────┐         │ │
 │  │  │ kubelet  │         │   │  │ kubelet  │         │ │
 │  │  └────┬─────┘         │   │  └────┬─────┘         │ │
 │  │       ▼               │   │       ▼               │ │
 │  │  ┌──────────┐         │   │  ┌──────────┐         │ │
 │  │  │ Container│         │   │  │ Container│         │ │
 │  │  │ Runtime  │         │   │  │ Runtime  │         │ │
 │  │  └──────────┘         │   │  └──────────┘         │ │
 │  │  ┌──────────┐         │   │  ┌──────────┐         │ │
 │  │  │kube-proxy│         │   │  │kube-proxy│         │ │
 │  │  └──────────┘         │   │  └──────────┘         │ │
 │  └───────────────────────┘   └───────────────────────┘ │
 └────────────────────────────────────────────────────────┘

2. Control Plane Components

The Control Plane manages the cluster. Its components can run on a single master node or be distributed across multiple nodes for high availability.

A. API Server (kube-apiserver)

• The Hub: The gateway to the cluster. All components (cli tools, worker nodes, controllers) communicate only with the API Server.
• Protocol: Exposes a RESTful API over HTTPS.
• Responsibilities: Authenticates callers, authorizes requests (RBAC), validates payloads, and writes changes directly to etcd. It is stateless.

B. etcd (State Storage)

• The Database: A highly available, distributed, key-value store. It holds the complete cluster state (the source of truth).
• Consensus: Uses the Raft consensus algorithm.
• Responsibilities: Storage only. No active logic runs here. Only the API Server has write/read access to etcd.

C. Scheduler (kube-scheduler)

• The Matchmaker: Watches for newly created pods that have no node assigned.
• Responsibilities: Runs the Filtering (Predicates) and Scoring (Priorities) phases to select the best worker node for the pod, then binds the pod to the node.

D. Controller Manager (kube-controller-manager)

• The Orchestrator: Runs the core control loops (reconciliation loops) that maintain the desired state.
• Built-in Controllers:
  • Node Controller: Monitors node status and handles outages.
  • Deployment Controller: Handles updates and rollouts.
  • EndpointSlice Controller: Links pods to Services.

3. Worker Node Components

A. kubelet

• The Agent: A native agent running on every node in the cluster. It communicates directly with the API Server.
• Responsibilities:
  • Receives PodSpecs from the API Server.
  • Communicates with the container runtime (via Container Runtime Interface - CRI) to spin up/tear down containers.
  • Monitors container health (probes) and reports metrics/status back to the API Server.

B. kube-proxy

• The Router: Runs on every node. It manages host network rules.
• Responsibilities: Translates Service IPs into actual Pod IPs by writing routing rules into host network tables (using iptables or IPVS).

C. Container Runtime (CRI)

• The Executor: The software that actually runs containers (e.g. containerd, CRI-O).

4. Operational Troubleshooting Commands

Verify the health of the Control Plane components (if self-hosted as static pods):

kubectl get pods -n kube-system -l tier=control-plane

Inspect kubelet logs on a worker node:

journalctl -u kubelet -n 100 --no-pager

View the detailed status of worker nodes:

kubectl get nodes -o wide

5. Architectural Interview Q&A

Q: If etcd crashes completely, does your application running inside the worker nodes go down? A: No. The worker nodes and the actual application containers will continue to run normally because kubelet and kube-proxy maintain local states (e.g., local iptables rules, local runtime container lists). However, the cluster control plane becomes completely non-functional: you cannot deploy new apps, modify configurations, scale pods, or handle node evictions until etcd is restored.

Q: Explain the exact flow of what happens when you run kubectl apply -f deployment.yaml. A:

1. Client-Side: kubectl validates the YAML structure and sends an HTTP POST request to the API Server.
2. API Server Authentication/Authorization: The API Server verifies the client certificate (auth) and checks RBAC permissions (authz).
3. etcd Write: The API Server writes the Deployment object to etcd.
4. Deployment Controller: The Deployment Controller (inside kube-controller-manager) detects the new Deployment, creates a ReplicaSet in etcd, and exits.
5. ReplicaSet Controller: The ReplicaSet Controller detects the ReplicaSet and creates Pod specs in etcd (with their nodeName fields empty).
6. Scheduler: The kube-scheduler detects the unscheduled Pods, evaluates worker nodes, selects a node, and writes the nodeName back to the Pod specs in etcd.
7. Kubelet: The kubelet on the selected worker node detects the scheduled Pod, instructs the local Container Runtime (via CRI) to start the containers, and reports the state back to the API Server.

6. Common Pitfalls

[!WARNING] etcd Latency and Disk I/O etcd is highly sensitive to disk I/O latency because it must write updates to disk synchronously. If you host etcd on slow HDDs or EBS volumes without dedicated IOPS, the cluster will experience frequent timeout errors, leader election loops, and control plane drops. Always back etcd with fast SSD storage.