SSL/TLS Fundamentals

TL;DR

• Asymmetric cryptography (public/private key pairs) enables secure handshakes; symmetric cryptography (shared session key) encrypts bulk data efficiently
• Certificates bind a public key to identity metadata (CN, SAN, expiry) and are signed by a Certificate Authority
• mTLS (mutual TLS) requires both server and client to present and validate certificates; default in Kubernetes control plane
• Subject Alternative Names (SAN) are the modern way to validate certificate hosts; CN is ignored by modern clients
• kubeadm certificates default to 1-year validity; expiry is the #1 cause of sudden control-plane failures
• Certificate validation must match the connection target (hostname or IP) against the SAN list; mismatches cause validation failures

Core Concepts

Public Key Infrastructure (PKI) Primitives

Asymmetric Key Pair — Consists of a private key (kept secret) and a mathematically related public key (shared freely). Data encrypted with the public key can only be decrypted with the private key. This asymmetry enables secure communication between parties who have never met.

Certificate — A public key bundled with identity metadata (Common Name, Subject Alternative Names, validity dates) and cryptographically signed by a Certificate Authority. A certificate proves the public key’s authenticity and identity.

Certificate Authority (CA) — A trusted entity whose signature on a certificate vouches for the certificate’s authenticity and the identity it represents. Kubernetes clusters maintain their own private internal CA; public CAs are not involved in control-plane component communication.

Chain of Trust — Transitive trust model: trusting a CA’s root certificate means trusting all certificates that CA has signed.

The TLS Handshake

TLS negotiates a secure connection through a multi-step handshake:

1. ClientHello — Client proposes supported TLS versions and cipher suites
2. ServerHello + Certificate — Server selects a cipher suite and sends its certificate (containing public key and CA signature)
3. Certificate Validation — Client verifies the server’s certificate: checks CA signature, validates hostname/IP against SAN, verifies expiry date
4. Key Exchange — Client and server perform a Diffie-Hellman key exchange to derive a shared symmetric session key
5. Finished — Both sides confirm successful handshake; encrypted application data now flows using the symmetric session key

Critical insight: Asymmetric encryption is used only during the handshake (to securely establish the shared key). All subsequent data is encrypted with symmetric encryption using that shared key, because symmetric crypto is orders of magnitude faster for bulk data.

Mutual TLS (mTLS)

Standard TLS is one-directional: the client verifies the server’s certificate. Mutual TLS adds a step where the server also requests and validates a certificate from the client, so both sides authenticate each other. Kubernetes control-plane components use mTLS exclusively: kubelet presents a certificate to the kube-apiserver, and the API server validates it.

Certificate Validation: The SAN Field

Modern TLS clients validate the connection target (hostname or IP address) strictly against the certificate’s Subject Alternative Names (SAN) field. The older Common Name (CN) field is functionally ignored.

A certificate with CN=kube-apiserver but SAN list [api.example.com, 10.0.0.5] will fail validation when connecting to any other hostname or IP, even though the certificate itself is valid and signed by a trusted CA.

kubeadm Certificate Lifecycle

kubeadm-generated certificates default to 1-year validity. After one year, all Kubernetes control-plane components fail mutual TLS validation simultaneously, and the cluster becomes completely unavailable. This silent, hard deadline catches many clusters off-guard.

Configuration and Operations Reference

Key Generation and Certificate Creation

# Generate a private key (2048-bit RSA)
openssl genrsa -out server.key 2048

# Generate a Certificate Signing Request (CSR)
openssl req -new -key server.key -out server.csr \
  -subj "/CN=kube-apiserver" \
  -addext "subjectAltName=DNS:kubernetes,DNS:kubernetes.default,IP:10.0.0.1"

# Self-sign the certificate (for testing/internal use)
openssl x509 -req -in server.csr \
  -signkey server.key -out server.crt \
  -days 365 \
  -extfile <(echo "subjectAltName=DNS:kube-apiserver,IP:10.0.0.1")

Inspecting Certificates

# View certificate contents and fields
openssl x509 -in server.crt -noout -text

# Check certificate expiry date
openssl x509 -in server.crt -noout -enddate

# Verify a certificate was signed by a specific CA
openssl verify -CAfile ca.crt server.crt

# Extract just the SAN field
openssl x509 -in server.crt -noout -text | grep -A1 "Subject Alternative Name"

Testing TLS Endpoints

# Connect to a TLS endpoint and display the certificate chain
openssl s_client -connect api.example.com:6443 -showcerts

# Test connection with hostname verification
openssl s_client -connect api.example.com:6443 -servername api.example.com

# Decode and inspect a Kubernetes TLS Secret
kubectl get secret my-tls -o jsonpath='{.data.tls\.crt}' | \
  base64 -d | openssl x509 -noout -text

kubeadm Certificate Management

# Check expiry dates of all control-plane certificates
kubeadm certs check-expiration

# Renew all expired/expiring certificates
kubeadm certs renew all

# Renew a specific certificate (e.g., kube-apiserver)
kubeadm certs renew apiserver

Common Pitfalls

Pitfall — Certificate SAN Missing the Connection Target Client connects to a valid certificate but receives a validation error despite the certificate being signed and unexpired.

• Why: The certificate’s SAN list doesn’t include the hostname or IP the client is using. Modern TLS clients validate strictly against SAN, not CN.
• Fix: Regenerate the certificate with the correct SAN list. For kubeadm API servers, use --apiserver-cert-extra-sans to add load-balancer IPs or additional hostnames.

Pitfall — Certificate Expiry Causes Sudden Cluster Failures Control-plane components all fail simultaneously after exactly 1 year; cluster becomes completely unavailable.

• Why: kubeadm certificates default to 1-year validity. After expiry, mutual TLS validation fails across all control-plane components.
• Fix: Monitor certificate expiry with kubeadm certs check-expiration. Renew certificates well before expiry (e.g., at 6 months) via kubeadm certs renew all or cluster upgrades.

Pitfall — Using CN Instead of SAN for Hostname Validation Assuming CN is sufficient for TLS hostname validation.

• Why: Modern clients ignore CN entirely and validate only against SAN. CN is deprecated for this purpose.
• Fix: Always include SAN when generating certificates; never rely on CN for validation.

Pitfall — Mixing Private and Public CAs in a Cluster Different components signed by different CAs, causing validation failures.

• Why: Components expect a single internal CA; mismatched CAs break the chain of trust.
• Fix: Ensure all internal Kubernetes components are signed by the same internal CA bundle (trust all internal certs or none).

Interview Questions

Q — Explain why TLS switches from asymmetric to symmetric encryption after the handshake. What is the performance trade-off?

Q — A client receives “certificate validation failed” for a valid, unexpired certificate. What is the likely cause and how do you diagnose it?

Q — What is the difference between CN (Common Name) and SAN (Subject Alternative Names)? Which does a modern TLS client check?

Q — Describe the mutual TLS (mTLS) handshake and explain why Kubernetes uses it for control-plane communication.

Q — How would you diagnose an expired control-plane certificate using kubeadm? What is the recovery procedure?

Q — Design a certificate rotation strategy for a 3-year-old kubeadm cluster that has never had its certificates rotated.

Q — A kubeadm cluster API server certificate includes SAN=[api.example.com, 10.0.0.5] but a client connects via a load-balancer IP 10.0.1.100 and fails. How do you fix this?

Q — Explain the purpose of a Certificate Authority (CA) and why Kubernetes clusters maintain their own internal CA.