TLS in Kubernetes

TL;DR

• kubeadm manages ~12 certificate/key pairs in /etc/kubernetes/pki/
• Certificate Signing Request (CSR) API enables secure certificate issuance without exposing the cluster CA’s private key
• kube-controller-manager (csrsigning controller) holds the CA private key and signs approved CSRs
• kubelet can auto-rotate certificates via rotateCertificates: true and serverTLSBootstrap: true
• kubeadm certs renew all updates files on disk but doesn’t restart static pods (manual restart required)
• Separate CAs exist for etcd (etcd/ca.crt) and aggregation layer (front-proxy-ca.crt)

Core Concepts

kubeadm Certificate Layout

A kubeadm cluster maintains approximately a dozen certificate/key pairs, each serving a specific purpose:

All files reside in /etc/kubernetes/pki/ on the control node.

Certificate Signing Request (CSR) API

Kubernetes exposes a native CertificateSigningRequest resource enabling secure certificate issuance without ever exposing the cluster CA’s private key to the requester.

Security model: A requester generates a key locally, creates a CSR (containing only their public key), submits it to the Kubernetes API, and an approver (human or automated controller) approves the request. The kube-controller-manager then signs the CSR using the CA private key (never exposed) and writes the resulting certificate to the CSR object.

This mechanism allows components like kubelet to bootstrap their own identity and tools like cert-manager to automate TLS certificate issuance cluster-wide.

Kubelet Certificate Rotation

Kubelets can automatically rotate their own client and serving certificates before expiry, using bootstrap tokens:

# kubelet configuration
rotateCertificates: true
serverTLSBootstrap: true

Kubelets submit kubelet-serving and kubelet-client CSRs for rotation. These are not auto-approved by default; an approver controller (or manual approval) is required for security.

Multiple CAs in a kubeadm Cluster

A single kubeadm cluster uses three separate CAs:

• Main cluster CA (ca.crt) — Signs API server, kubelet, and control-plane component certificates
• etcd CA (etcd/ca.crt) — Separate CA for etcd’s internal mutual TLS, isolated from the main cluster CA
• Front-proxy CA (front-proxy-ca.crt) — Separate CA for aggregation layer components (metrics-server, extension API servers)

This multi-CA design compartmentalizes trust: etcd operates independently, and the aggregation layer is sandboxed from direct cluster CA access.

Static Pod Certificate Loading

All kubeadm control-plane components (kube-apiserver, kube-scheduler, kube-controller-manager, etcd) run as static pods. These pods load certificates into memory at startup time. Renewing certificate files on disk does not restart the pods; they continue using the old in-memory certificates until explicitly restarted.

Configuration and Operations Reference

Inspecting kubeadm Certificates

# List all kubeadm-managed certificates and their expiry dates
kubeadm certs check-expiration

# List certificate files in the PKI directory
ls -la /etc/kubernetes/pki/

# Inspect a certificate file directly
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text

# Check which CA signed a certificate
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -issuer

# Check certificate expiry
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -enddate

Certificate Signing Request Workflow

# 1. Generate a private key and CSR locally (on the client machine)
openssl genrsa -out jane.key 2048
openssl req -new -key jane.key -out jane.csr \
  -subj "/CN=jane-dev/O=developers"

# 2. Create a CertificateSigningRequest resource
cat <<EOF | kubectl apply -f -
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: jane-dev-csr
spec:
  signerName: kubernetes.io/kube-apiserver-client
  usages:
  - digital signature
  - key encipherment
  - client auth
  request: $(cat jane.csr | base64 | tr -d '\n')
EOF

# 3. Check CSR status
kubectl get csr jane-dev-csr
kubectl describe csr jane-dev-csr

# 4. Approve the CSR (only approvers can do this)
kubectl certificate approve jane-dev-csr

# 5. Extract the signed certificate
kubectl get csr jane-dev-csr -o jsonpath='{.status.certificate}' | base64 -d > jane.crt

Renewing kubeadm Certificates

# Check expiry dates
kubeadm certs check-expiration

# Renew all certificates
kubeadm certs renew all

# Renew specific certificate (e.g., apiserver)
kubeadm certs renew apiserver

# AFTER renewal, restart static pods to pick up new certs
# Option 1: Remove and restore manifests
sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/
# wait for pod to be deleted by kubelet
sleep 10
sudo mv /tmp/kube-apiserver.yaml /etc/kubernetes/manifests/

# Option 2: Kill the pod directly (kubelet will restart it)
kubectl delete pod -n kube-system kube-apiserver-<node-name>

Common Pitfalls

Pitfall — Certificates Renewed But Components Still Use Old Certs Running kubeadm certs renew all but control-plane components continue using old certificates.

• Why: Static pods loaded the old certificates into memory at startup. Renewing files on disk doesn’t restart the pods; they continue running with cached old certs.
• Fix: After renewal, restart the static pods by deleting their manifest files or restarting the kubelet, forcing them to reload the new certs.

Pitfall — Troubleshooting TLS Failures in the Wrong CA TLS handshakes fail between API server and etcd, but inspection of the main ca.crt shows everything correct.

• Why: etcd uses a separate CA (etcd/ca.crt). The main cluster CA is irrelevant to etcd connections.
• Fix: Always identify which specific connection is failing, then check the correct CA. For etcd connections, inspect etcd/ca.crt. For API server to kubelet connections, check the main ca.crt.

Pitfall — CSR Stays Pending Forever Submitted a CSR but it never gets approved or signed.

• Why: CSRs must be explicitly approved; no auto-approval exists for user CSRs by default. Only kubelet-serving CSRs might have automatic approval (if a controller is deployed).
• Fix: Check RBAC permissions for who can approve CSRs. Manually approve with kubectl certificate approve or deploy an automating approver controller.

Pitfall — Exposing CA Private Key During CSR Submission Assuming the CSR API requires sharing the CA private key.

• Why: The CSR API is designed specifically to avoid private key exposure. Only the requester’s public key is submitted.
• Fix: CSRs are inherently safe; the CA private key is held only by the kube-controller-manager and never exposed.

Interview Questions

Q — Explain how the Certificate Signing Request (CSR) API allows secure certificate issuance without exposing the cluster CA’s private key.

Q — Which kubeadm component holds the cluster CA private key and performs CSR signing?

Q — After running kubeadm certs renew all, control-plane components are still using old certificates. What’s missing and how do you fix it?

Q — Describe the differences between the main cluster CA, the etcd CA, and the front-proxy CA. Why does kubeadm use multiple CAs?

Q — How does kubelet automatically rotate its serving and client certificates? Are these CSRs auto-approved by default?

Q — A TLS connection from the API server to etcd is failing with a certificate validation error. Which CA file would you check first?

Q — Design a CSR workflow for onboarding a new developer with kubectl access.

Q — What happens to kubelet certificates on worker nodes added to a kubeadm cluster post-creation? How does certificate rotation work?