CoreDNS

TL;DR

• CoreDNS is the default cluster DNS server; every pod’s /etc/resolv.conf points to the kube-dns Service ClusterIP
• Kubernetes DNS naming: service-name.namespace.svc.cluster.local for normal Services, pod IPs directly for headless Services
• ndots:5 in pod /etc/resolv.conf causes external domain lookups to retry against cluster search domains first, adding latency
• Headless Services (clusterIP: None) resolve to individual pod IPs via A/SRV records instead of a single virtual ClusterIP
• CoreDNS ConfigMap (Corefile) is live-reloaded; syntax errors can break cluster DNS immediately; test changes carefully
• Forward external queries upstream via forward . /etc/resolv.conf in Corefile; caching reduces upstream load

Core Concepts

Kubernetes DNS Naming Convention

For a Service payments-api in namespace payments:

• Fully qualified: payments-api.payments.svc.cluster.local → resolves to ClusterIP
• Short form (same namespace): payments-api → search domains auto-append suffix
• Cross-namespace: payments-api.payments → minimal FQDN needed

Headless Services (clusterIP: None) resolve to individual pod IPs directly via A records (or SRV records per pod), enabling StatefulSets to give each replica a stable DNS name:

• web-0.nginx.default.svc.cluster.local
• web-1.nginx.default.svc.cluster.local

Pod /etc/resolv.conf Configuration

Every pod receives resolver configuration injected by kubelet:

nameserver 10.96.0.10                                    # kube-dns Service ClusterIP
search <namespace>.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

ndots:5 latency trap: Any name with fewer than 5 dots is tried against every search domain first before absolute lookup. External domains like api.stripe.com (2 dots) generate wasted queries against cluster domains before finally succeeding as absolute.

CoreDNS Corefile Structure

.:53 {
    errors
    health
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {
       pods insecure
       fallthrough in-addr.arpa ip6.arpa
    }
    prometheus :9153
    forward . /etc/resolv.conf {
       max_concurrent 1000
    }
    cache 30
    loop
    reload
    loadbalance
}

Key directives:

• kubernetes: Handles cluster-internal DNS lookups
• forward .: Sends upstream anything not resolved cluster-internally
• cache 30: Caches responses for 30 seconds
• reload: Auto-reloads config on ConfigMap change (risky in production)

Comparison Table

Command Reference

Test DNS from Inside Cluster

# Deploy temporary debug pod
kubectl run -it debug --image=alpine -- sh

# Inside the pod:
nslookup my-service.default.svc.cluster.local
dig my-service.default.svc.cluster.local
cat /etc/resolv.conf

Check CoreDNS Configuration

# View Corefile
kubectl get configmap coredns -n kube-system -o yaml | grep Corefile -A 50

# Edit Corefile
kubectl edit configmap coredns -n kube-system

# View CoreDNS pod logs
kubectl logs -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system deployment/coredns

Diagnose DNS Issues

# Check if CoreDNS pods are running and healthy
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl describe pod -n kube-system <coredns-pod>

# Verify kube-dns Service endpoints
kubectl get endpoints -n kube-system kube-dns
kubectl get service -n kube-system kube-dns

# Check pod DNS configuration
kubectl exec <pod> -- cat /etc/resolv.conf

# Resolve a service from inside pod
kubectl exec <pod> -- nslookup kubernetes.default

Create Headless Service

apiVersion: v1
kind: Service
metadata:
  name: mysql
  namespace: default
spec:
  clusterIP: None
  selector:
    app: mysql
  ports:
  - port: 3306
    targetPort: 3306

Common Pitfalls

Pitfall: ndots:5 Causes External DNS Latency External domain lookups from pods feel slow (takes 10+ seconds for single DNS query).

• Why: ndots:5 setting in pod /etc/resolv.conf causes resolver to try every short name against all cluster search domains first. External names like api.stripe.com (2 dots) get tried against <namespace>.svc.cluster.local, svc.cluster.local, cluster.local before final absolute lookup succeeds.
• Fix: Use fully-qualified external domains in application code (e.g., api.stripe.com. with trailing dot). Reduce ndots value if using tools that allow it (advanced, not recommended for production).

Pitfall: CoreDNS Syntax Error Breaks Cluster DNS Instantly Editing CoreDNS ConfigMap directly with a typo; cluster-wide DNS stops working within seconds.

• Why: CoreDNS ConfigMap has reload plugin enabled, which auto-reloads on any ConfigMap change. Syntax errors in Corefile cause immediate reload failure; DNS becomes unavailable.
• Fix: Test Corefile syntax in isolated environment first. Use kubectl rollout to quickly revert ConfigMap on syntax error. Consider disabling reload plugin for production stability.

Pitfall: Intermittent DNS Failures with Identical Pod Specs Some pods resolve services fine, others fail intermittently.

• Why: CoreDNS runs as a Deployment with multiple replicas behind the kube-dns Service. If one replica is unhealthy or slow, some requests hit that replica and fail/timeout; others hit healthy replicas and succeed.
• Fix: Ensure CoreDNS pods are healthy: kubectl get pods -n kube-system -l k8s-app=kube-dns. Scale CoreDNS appropriately for cluster size. Monitor CoreDNS pod logs for errors.

Pitfall: Headless Service Returns Multiple IPs Unexpectedly Creating a headless Service, querying its DNS, and getting multiple pod IPs instead of expecting a single ClusterIP.

• Why: This is correct behavior for headless Services—DNS returns all backend pod IPs (via A or SRV records). Clients must handle multiple IPs; load balancing is client-side, not kube-proxy-based.
• Fix: Ensure applications are designed to handle multiple target IPs for headless Services. For single-target scenarios, use a regular Service instead.

Interview Questions

Q — How does Kubernetes DNS naming work? Why is the fully-qualified domain name necessary for cross-namespace communication? Kubernetes DNS follows the pattern service-name.namespace.svc.cluster.local. Every pod’s /etc/resolv.conf has search directives that auto-append suffixes to short names. Within the same namespace, my-service resolves via search domains appending .my-namespace.svc.cluster.local. Cross-namespace requires explicit namespace in the name (e.g., my-service.other-namespace) or full FQDN, because search domains only append namespace-scoped suffixes; they don’t try all namespaces.

Q — What is the ndots:5 setting in pod /etc/resolv.conf and why does it cause slow external DNS lookups? ndots:5 instructs the resolver to try any name with fewer than 5 dots as a relative name (appending search domains) before trying it as absolute. External domains like api.stripe.com (2 dots) get tried against all cluster search domains first (<ns>.svc.cluster.local, svc.cluster.local, etc.), generating wasted queries, before the absolute lookup finally succeeds. Total latency can be seconds. Fix: Use fully-qualified external domains with trailing dots (e.g., api.stripe.com.) or redesign to avoid short external names if possible.

Q — How does a headless Service differ from a regular Service in terms of DNS? Regular Service DNS resolves to a single virtual ClusterIP (assigned by Kubernetes, load-balanced by kube-proxy). Headless Service DNS (clusterIP: None) returns individual backend pod IPs directly (via A or SRV records), enabling clients to choose specific backends. StatefulSets use headless Services to give each pod a stable, individually-addressable DNS name (e.g., web-0.nginx, web-1.nginx). Load balancing is client-side, not enforced by kube-proxy.

Q — Why is editing the CoreDNS ConfigMap in production risky? How would you safely update DNS configuration? CoreDNS ConfigMap has the reload plugin enabled, auto-reloading whenever the ConfigMap changes. A syntax error in the Corefile causes immediate reload failure, breaking cluster-wide DNS within seconds. Multiple pods become unable to resolve Service names, cascading to widespread application failures. Safe approach: test Corefile changes in a staging cluster first, use a version control system for ConfigMap changes, enable alerting on CoreDNS pod restarts, practice quick rollback via kubectl rollout or reverting the ConfigMap to previous version.