Kubernetes Networking

TL;DR

• Kubernetes itself implements almost no networking; it delegates to CNI plugins (Flannel, Calico, AWS VPC CNI)
• Three fundamental requirements: unique pod IPs, pod-to-pod communication without NAT cluster-wide, kubelet-to-pod communication
• Pod network CIDR and Service cluster IP range are completely independent CIDR pools configured at cluster bootstrap
• Service ClusterIP is a virtual IP that exists only as kube-proxy DNAT rewrite targets; no actual network interface
• kube-proxy modes: iptables (legacy, O(n) lookup), IPVS (production, O(1) lookup), userspace (deprecated)
• CNI implementations vary: overlay networks (Flannel VXLAN, adds latency), BGP routing (Calico, no encapsulation), cloud-native VPC CNI (real IPs from VPC)

Core Concepts

The Kubernetes Networking Model

Kubernetes mandates three non-negotiable requirements; implementation is entirely CNI plugin’s responsibility:

1. Every pod receives a unique IP address (no NAT between pods on same node)
2. Pod-to-pod communication works cluster-wide without NAT, regardless of node boundaries
3. Agents on a node (kubelet) communicate with pods on that node directly

Kubernetes never specifies how—only what. Different CNI plugins achieve this differently.

Pod Network CIDR vs. Service Cluster IP Range

These are completely independent CIDR pools configured at cluster bootstrap:

Pod Network CIDR: (--pod-network-cidr in kubeadm) Allocates real IPs to pods. Each node gets a subnet carved from this range; CNI assigns pods IPs from their node’s subnet. Example: 10.244.0.0/16 (pods), with nodes getting /24 subnets like 10.244.0.0/24, 10.244.1.0/24.

Service Cluster IP Range: (--service-cidr in kubeadm) Allocates virtual IPs to Services. These IPs never appear on network interfaces; they exist only as kube-proxy rewrite targets. Example: 10.96.0.0/12.

Critical: These ranges must not overlap with each other or with node network.

kube-proxy Modes

iptables: Uses Linux kernel iptables rules to DNAT Service traffic to backend pods. Linear lookup time (O(n) rules per Service), high CPU at scale. Legacy default.

IPVS: Uses Linux IPVS (IP Virtual Server) kernel module for load balancing. Constant-time lookup (O(1)) regardless of Service count. Production recommended. More scalable, better performance.

userspace: Proxy runs in userspace, forwarding traffic through user-space processes. Deprecated; high overhead.

CNI Implementation Approaches

Overlay Networks (Flannel VXLAN): Wraps pod traffic in VXLAN encapsulation; traverses existing network transparently. Simple deployment, slight latency overhead from encapsulation.

BGP Routing (Calico): Advertises pod CIDR routes via BGP; no encapsulation. Requires BGP-capable infrastructure; zero encapsulation overhead.

Cloud-native VPC CNI (AWS VPC CNI, Azure CNI): Assigns pods real IPs from VPC/subnet space; pods become first-class network citizens. Simplifies integration with security groups, no encapsulation overhead, but consumes real IPs per pod.

Comparison Table

Command Reference

Check CNI Installation

# List CNI plugins on nodes
kubectl get pods -n kube-system | grep -Ei "calico|flannel|cilium|aws-node"

# Check CNI config on node
kubectl debug node/<node-name> -- cat /etc/cni/net.d/*

# Check kube-proxy mode
kubectl get daemonset -n kube-system kube-proxy -o yaml | grep "mode" | head -5

Inspect Service Networking

# List Services and their Cluster IPs
kubectl get svc -o wide

# Check kube-proxy rules (iptables mode)
kubectl debug node/<node-name> -- iptables -t nat -L -n | grep <service-cluster-ip>

# Check kube-proxy rules (IPVS mode)
kubectl debug node/<node-name> -- ipvsadm -Ln

Test Pod-to-Pod Networking

# Deploy test pods
kubectl run -it client --image=alpine -- sh
kubectl run -it server --image=alpine -- sleep 3600

# From client pod:
ping <server-pod-ip>
nc -zv <server-pod-ip> 80

Diagnose Cross-Node Communication Failure

# Verify node-to-node routing
kubectl debug node/<node-1> -- ip route

# Check if pods on different nodes can reach each other
kubectl exec <pod-on-node-1> -- ping <pod-ip-on-node-2>

# Check CNI pod logs
kubectl logs -n kube-system -l app=<cni-name>

Common Pitfalls

Pitfall: Pod Network CIDR Overlaps with Node Network Cluster pods cannot route correctly; pods on same node can talk, but cross-node fails.

• Why: Routing decisions become ambiguous when pod CIDR overlaps with existing node network. Kernel cannot distinguish between “destination is pod on remote node” and “destination is node itself.”
• Fix: Choose non-overlapping CIDR ranges at cluster bootstrap. Verify: kubectl get nodes -o wide shows node IPs; --pod-network-cidr must not include those ranges.

Pitfall: Same-Node Pods Communicate, Cross-Node Fails Pods work fine on the same node but cannot reach pods on other nodes.

• Why: Problem is in CNI’s cross-node routing/overlay mechanism, not in kube-proxy. Either CNI is not installed, or its networking layer is broken (overlay network down, BGP routes not advertised, etc.).
• Fix: Verify CNI pods are running: kubectl get pods -n kube-system. Check CNI logs for errors. Restart CNI if misconfigured.

Pitfall: Scaling Services Causes kube-proxy CPU Spikes (iptables mode) Adding more Services causes kube-proxy CPU usage to grow linearly; rule evaluation becomes slow.

• Why: iptables mode uses linear rule lookups (O(n)). Each new Service adds more rules; lookup time increases proportionally.
• Fix: Switch kube-proxy to IPVS mode via kubeadm init --proxy-mode=ipvs or update existing cluster’s kube-proxy ConfigMap and DaemonSet.

Pitfall: Service ClusterIP Not Responding Creating a Service, getting its ClusterIP, but traffic doesn’t reach backend pods.

• Why: ClusterIP is virtual; it only exists as kube-proxy rewrite rules. If kube-proxy is not running, or pod selector is wrong (backend pods don’t match service selector), traffic has nowhere to go.
• Fix: Verify kube-proxy is running: kubectl get ds -n kube-system kube-proxy. Verify backend pods match Service selector: kubectl get pods -l <selector>. Check endpoint: kubectl get endpoints <service-name>.

Interview Questions

Q — Kubernetes delegates all networking to the CNI. What are the three fundamental requirements that every CNI plugin must satisfy? Every pod must have a unique IP address (no pod-to-pod NAT on same node). Pod-to-pod communication must work cluster-wide without NAT, regardless of node boundaries—a pod on node A must reach a pod on node B at that pod’s IP directly. Agents on a node (kubelet) must communicate with all pods on that node. Kubernetes itself enforces none of this; the CNI plugin is entirely responsible for implementation.

Q — What is a Service ClusterIP mechanically? How does kube-proxy make it work? A ClusterIP is a virtual IP that does not exist on any network interface. When a pod sends traffic to a Service ClusterIP, kube-proxy (running on every node) rewrites the destination IP from the virtual ClusterIP to a real backend pod IP via DNAT. Different kube-proxy modes implement this differently: iptables uses kernel iptables rules for rewriting, IPVS uses kernel IPVS load balancing. The ClusterIP exists only in the minds of rewrite rules.

Q — Pod network CIDR and Service cluster IP range are configured completely independently. Why must they not overlap? Pod network CIDR allocates real IPs to pods; Service cluster IP range allocates virtual IPs to Services. If they overlap, the kernel’s routing decisions become ambiguous: is a destination IP a pod on a remote node, or a Service? Routing fails. Additionally, kube-proxy’s rewrite rules assume these ranges are distinct so it can distinguish between regular pod traffic and Service traffic that needs DNAT rewriting.

Q — Why does kube-proxy in iptables mode struggle at scale, and how does IPVS fix it? iptables mode creates one or more iptables rules per Service. Rule lookup is linear—the kernel walks rules sequentially until finding a match. At scale (hundreds of Services), rule count grows dramatically, and each packet traversal becomes slow (O(n) time). IPVS replaces this with kernel-native load balancing using hash tables, giving constant-time lookup (O(1)) regardless of Service count. IPVS is production-standard for large clusters.