Logging & Monitoring

TL;DR

• Kubernetes provides no built-in log aggregation or long-term metrics storage; kubectl logs reads node files and evaporates when pods are deleted
• metrics-server powers kubectl top and HPA CPU/memory scaling; only provides metrics.k8s.io API (no custom metrics)
• Custom metrics (queue depth, RPS) require separate scrapers and custom metrics API server (Prometheus + kube-custom-metrics or similar)
• Standard observability stack: Prometheus (metrics) → Grafana (visualization), Promtail/Filebeat → Loki/ELK (logs)
• kubectl logs --previous retrieves logs from previous container instance (if still on node); once pod deleted, logs are gone forever without external storage
• Production observability requires external components; kubectl logs is useful only for immediate troubleshooting, not forensics

Core Concepts

Kubernetes Observability Limitations

Kubernetes itself provides no log aggregation, no metrics retention, no persistence. kubectl logs reads directly from node’s /var/log/pods/ directory; logs disappear when pods are deleted or nodes log-rotated. kubectl top requires metrics-server to aggregate kubelet metrics in-memory, with no long-term storage.

metrics-server

Aggregates CPU and memory metrics from kubelet and exposes them via metrics.k8s.io API. Installed by default on most managed clusters.

• Powers kubectl top
• Enables HPA scaling on CPU/memory metrics
• Does NOT provide custom metrics; only CPU/memory
• Metrics are short-lived (in-memory only, no database)

Custom Metrics API

To scale on custom metrics (queue depth, RPS, latency), implement separate custom metrics API (e.g., Prometheus adapter, Stackdriver adapter). HPA queries custom.metrics.k8s.io for custom metrics.

Example HPA with custom metric:

metrics:
- type: Pods
  pods:
    metric:
      name: http_requests_per_second
    target:
      averageValue: "1000"

Standard Observability Stack

Metrics Path: Application → Prometheus scraper → Prometheus database → Grafana dashboard

Logs Path: Application (stdout/stderr) → Filebeat/Promtail → Loki/ELK → Grafana/Kibana

Traces Path: Application instrumentation → Jaeger/Tempo collector → storage → Grafana Loki/Jaeger UI

Grafana Role

Visualization layer for multiple backends:

• Queries Prometheus for metrics
• Queries Loki/ELK for logs
• Queries Jaeger for traces
• Single dashboard correlates all three

Comparison Table

Command Reference

Metrics & Health

# View current CPU/memory of pods
kubectl top nodes
kubectl top pods -A

# Check metrics-server health
kubectl get deployment metrics-server -n kube-system

# Query Prometheus directly (if port-forward)
kubectl port-forward -n monitoring svc/prometheus 9090:9090
# Visit http://localhost:9090

Logging

# View pod logs (current container)
kubectl logs <pod> [-c <container>]

# View logs from previous container instance
kubectl logs --previous <pod>

# Stream logs (follow)
kubectl logs -f <pod>

# View logs for all pods matching label
kubectl logs -l app=myapp

# View last N lines
kubectl logs <pod> --tail=100

# Retrieve logs from X time ago
kubectl logs <pod> --since=10m

Log Aggregation Setup

# Install Loki/Promtail via Helm
helm repo add grafana https://grafana.github.io/helm-charts
helm install loki grafana/loki-stack -n monitoring

# Deploy ELK stack alternative
helm install elasticsearch elastic/elasticsearch
helm install kibana elastic/kibana

HPA with metrics-server

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: cpu-scaler
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 80

Common Pitfalls

Pitfall: Relying on kubectl logs for Post-Incident Forensics Incident occurs, pod is deleted/rescheduled, logs are gone; cannot investigate root cause.

• Why: kubectl logs reads from node’s local log files (typically stored 5-30 days per node policy). Once pod is deleted, kubelet garbage-collects the pod directory. Logs are permanently lost without external aggregation.
• Fix: Deploy external log aggregation (Loki, ELK, cloud logging) immediately. Send all logs to centralized storage for retention. Set up alerting to capture logs before pod deletion.

Pitfall: Using metrics-server for Custom Metrics (Queue Depth, RPS) HPA configured to scale on custom metrics; scaling never triggers.

• Why: metrics-server only implements metrics.k8s.io API (CPU/memory). Custom metrics require custom.metrics.k8s.io API, which needs a separate custom metrics adapter (Prometheus adapter, cloud provider adapter, etc.).
• Fix: Deploy Prometheus + custom metrics adapter. Configure HPA to query custom.metrics.k8s.io. Test: kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/ | jq.

Pitfall: kubectl top Shows “unknown” on Nodes kubectl top nodes shows unknown metrics; metrics-server not aggregating.

• Why: metrics-server is either not installed, or kubelet metrics endpoint is not accessible (RBAC, network, TLS issues).
• Fix: Verify metrics-server is running: kubectl get deployment metrics-server -n kube-system. Check logs: kubectl logs -n kube-system deployment/metrics-server. Verify kubelet’s metrics endpoint: kubectl debug node/<node> -- curl --cacert /etc/kubernetes/pki/ca.crt https://127.0.0.1:10250/metrics.

Pitfall: Logs from Previous Container Lost Pod restarts; kubectl logs --previous returns “no previous logs found” despite expecting logs.

• Why: --previous retrieves logs from the immediately prior container instance. Once a new container is created and logs are written to new instance, old logs become inaccessible. If the pod is deleted or rescheduled to a different node, previous logs may already be garbage-collected by the original node’s kubelet.
• Fix: Set up external log aggregation to capture logs before container restart. Use kubectl logs during container’s running lifecycle, not after restart.

Pitfall: Prometheus Database Grows Unboundedly Prometheus disk usage grows daily; storage runs out.

• Why: Prometheus stores metrics time-series by default indefinitely (or until manually pruned). Without retention policy, old data accumulates. High-cardinality metrics (per-pod, per-container) multiplies storage.
• Fix: Set --storage.tsdb.retention.time=15d in Prometheus args. Reduce cardinality via metric relabeling. Use Prometheus remote storage for long-term archival.

Interview Questions

Q — Kubernetes provides no built-in log aggregation or long-term metrics storage. Why is external observability essential for production? Kubernetes’ kubectl logs reads directly from node’s local log files; once a pod is deleted, rescheduled to a different node, or logs are rotated, logs are permanently lost. Similarly, metrics-server stores metrics in-memory only (no persistence). In production, incidents require investigation after pods are gone; without external log aggregation and metrics storage, logs and metrics are unavailable for forensics. External stacks (Loki, ELK, Prometheus) capture logs and metrics to persistent databases, enabling post-incident analysis.

Q — What is the difference between metrics-server and a custom metrics API? When would you use each? metrics-server aggregates CPU and memory metrics from kubelet and exposes the metrics.k8s.io API; it powers kubectl top and enables HPA to scale on CPU/memory. Custom metrics API (custom.metrics.k8s.io) is implemented by a separate component (Prometheus adapter, cloud provider adapter) and enables HPA scaling on application-level metrics like queue depth, RPS, or latency. Use metrics-server for built-in resource metrics; deploy custom metrics adapter for application-specific scaling logic.

Q — How does kubectl logs --previous work and when is it useful? kubectl logs --previous retrieves logs from the immediately prior container instance (if the container was restarted but the pod still exists on the same node). This is useful for debugging why a recently-restarted container was killed (OOMKilled, CrashLoopBackOff, etc.). Once the pod is deleted or rescheduled to a different node, previous logs are garbage-collected and become inaccessible. This is why external log aggregation is essential—it captures logs before they’re lost.

Q — What role does Grafana play in the observability stack, and why is it not sufficient alone? Grafana is purely a visualization layer; it has no data storage. It queries multiple backends: Prometheus for metrics, Loki/ELK for logs, Jaeger for traces. Grafana itself doesn’t collect or store data. A complete observability stack requires both the data sources (Prometheus, Loki, Jaeger) and Grafana to visualize them. Grafana’s value is correlating data from multiple sources on a single dashboard; using only Grafana without backends produces no data to visualize.