Prometheus & PromQL (Deep Dive)

TL;DR

• Pull model: Prometheus scrapes /metrics endpoints on an interval, rather than applications pushing metrics. Advantages: overload-resistant, easy local debugging.
• Metric types: Counter (only increases), Gauge (up/down), Histogram (percentile buckets), Summary (pre-calculated quantiles).
• rate() calculates per-second average over a range. Handles counter resets on pod restarts. Use for alerting. irate() is volatile; use for high-resolution debugging only.
• Histograms enable percentile queries: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) for P99 latency.
• Cardinality explosion: unbounded labels (user_id, client_ip) create unique time series per value. Causes memory exhaustion. Fix: use metric_relabel_configs to drop high-cardinality labels.
• Alertmanager deduplicates, groups, routes, and silences alerts. Never send 50 alerts for 50 pod crashes—group into one notification.

Pull vs. Push Architecture

Aspect	Pull (Prometheus)	Push (StatsD, etc.)
Flow	Prometheus scrapes /metrics endpoints periodically	App pushes metrics to central agent
Overload Safety	Overloaded service: scrape times out; up metric = 0	Overloaded service: overwhelmed pushing metrics
Debugging	Run local Prometheus on laptop pointing to endpoints	Need to reconfigure app for local debugging
Service Discovery	Kubernetes API auto-discovers Pods by labels	Manual config per host/service
Data Freshness	Scrape interval (15s–1m typical)	Near real-time (but lossy)

Prometheus Architecture

APPLICATIONS (expose :8080/metrics)
        ↓
SERVICE DISCOVERY (Kubernetes API, static files, etc.)
        ↓
SCRAPER (pulls metrics every 15s)
        ↓
TSDB (stores time-series data)
        ↓
ALERTING ENGINE (evaluates rules)
        ↓
ALERTMANAGER (deduplicates, routes, groups)
        ↓
ALERTING TARGETS (Slack, PagerDuty, email)
        ↓
QUERYING & VISUALIZATION (PromQL, Grafana)

Metric Types & Usage

Type	Behavior	Example	PromQL Function
Counter	Only increases (or resets on restart)	http_requests_total, errors_total	rate(), increase()
Gauge	Goes up and down	memory_usage_bytes, active_connections	Direct query, rate() for trend
Histogram	Buckets for distributions	http_request_duration_seconds_bucket	histogram_quantile()
Summary	Pre-calculated quantiles	Deprecated; use Histogram instead	Direct query (slower to query)

Core PromQL Concepts

Instant Vector

Single data point per time series at a specific timestamp. Graphable directly.

http_requests_total{job="api", instance="server1"}  →  5000
http_requests_total{job="api", instance="server2"}  →  3500

Range Vector

Set of data points over time range. Cannot be graphed; must use aggregation function.

http_requests_total{job="api"}[5m]
→ [(t-5m, 4900), (t-4m, 4950), ..., (t, 5000)]

rate() Function

Per-second average rate of increase over range. Handles counter resets.

rate(http_requests_total[5m])
→ calculates: (value_now - value_5m_ago) / 300_seconds
→ result: ~1.67 req/sec average over last 5 minutes

irate() Function

Per-second rate based on last two data points. Highly volatile; use for zooming, not alerting.

irate(http_requests_total[5m])
→ calculates: (value_now - value_last_scrape) / scrape_interval
→ result: sharp spikes visible; noisy

Histogram Percentile Queries

To calculate P99 latency from histogram buckets:

histogram_quantile(
  0.99,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le)
)

Breaks down:

• http_request_duration_seconds_bucket = histogram buckets (le = “less than or equal” boundary)
• rate(...[5m]) = requests/second in each bucket over 5m
• sum(...) by (le) = cumulative sum across all instances, preserve bucket boundaries
• histogram_quantile(0.99, ...) = 99th percentile boundary

Result: P99 latency in seconds.

Alertmanager Configuration & Workflow

global:
  slack_api_url: 'https://hooks.slack.com/services/...'

route:
  receiver: 'default-receiver'
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
  - match:
      severity: page
    receiver: 'pagerduty-receiver'
    continue: true
  - match:
      severity: ticket
    receiver: 'slack-receiver'

receivers:
- name: 'default-receiver'
  slack_configs:
  - channel: '#alerts'
    title: 'Alert: {{ .GroupLabels.alertname }}'
    text: '{{ range .Alerts }}{{ .Labels.instance }}: {{ .Annotations.description }}{{ end }}'

- name: 'pagerduty-receiver'
  pagerduty_configs:
  - service_key: 'pagerduty-key-123'
    description: '{{ .GroupLabels.alertname }}'

inhibit_rules:
- source_match:
    severity: page
  target_match:
    severity: ticket
  equal: ['alertname', 'service']
  # Page-severity alerts inhibit ticket alerts for same service

Alertmanager Concepts:

• Deduplication: If 50 pod replicas crash, 50 alerts fire, but Alertmanager groups them into one notification.
• Grouping: Group by label (e.g., cluster, service). All alerts for the same service in one message.
• Routing: Different rules for different severities. PAGE goes to PagerDuty; TICKET goes to Slack.
• Silences: Temporarily mute alerts during maintenance windows.
• Inhibition: If critical alert fires, suppress non-critical alerts for same service.

PromQL Query Examples

Query	Purpose
rate(http_requests_total[5m])	HTTP requests per second (averaged over 5 min)
sum(rate(http_requests_total[5m])) by (job)	Per-job request rate
topk(5, rate(http_requests_total[5m]))	Top 5 services by request rate
http_requests_total - http_requests_total offset 5m	Requests in last 5 minutes (delta)
100 * (rate(http_errors_total[5m]) / rate(http_requests_total[5m]))	Error rate as percentage
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))	P99 latency
predict_linear(disk_free_bytes[1h], 3600)	Predict free disk space in 1 hour

Configuration Reference: Prometheus Scrape Config

global:
  scrape_interval: 15s
  scrape_timeout: 10s
  evaluation_interval: 15s
  external_labels:
    cluster: 'production'

scrape_configs:
- job_name: 'kubernetes-pods'
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  # Only scrape pods with prometheus=true annotation
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus]
    action: keep
    regex: 'true'
  # Extract pod name and namespace
  - source_labels: [__meta_kubernetes_pod_name]
    action: replace
    target_label: pod
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: namespace
  # Use pod port 8080
  - source_labels: [__meta_kubernetes_pod_container_port_number]
    action: keep
    regex: '8080'

metric_relabel_configs:
  # Drop high-cardinality labels
  - source_labels: [__name__]
    regex: 'request_duration_seconds'
    action: drop
  # Rename labels
  - source_labels: [pod_name]
    action: replace
    target_label: pod
  # Drop specific high-cardinality label values
  - source_labels: [user_id]
    action: drop

Common Pitfalls

Pitfall	Why It Happens	Fix
irate() in alerting rules	irate() shows last-scrape-interval rate—very noisy, micro-spikes. Causes false positive alerts.	Use rate() for alerting. irate() only for high-resolution zoomed graphs during debugging. rate() averages over range; irate() is instantaneous.
Cardinality explosion	Unbounded labels (user_id, client_ip, request_id) create unique time series per value. Million users = million time series. Prometheus RAM exhausted.	Never use unbounded labels. Use metric_relabel_configs to drop high-cardinality labels before ingestion. Code review: enforce label discipline.
Counter resets ignored	Comparing raw counter values across pod restarts. On restart, counter resets to 0. Query returns negative values.	Always use rate() or increase() for counters. These functions handle resets automatically. Never use raw counter in alerting.
Histogram quantiles wrong	Forgetting to sum buckets across instances. Query returns per-instance quantile (tiny).	Always sum(...) by (le) before histogram_quantile(). The le label (bucket boundary) must be preserved.
Scrape timeout too short	5s scrape timeout on slow application. Scrapes timeout; up metric = 0; false alert.	Set scrape_timeout generously (30–60s). Monitor scrape duration. Alert on up == 0. If slow, increase timeout instead of alert threshold.
No instance labels	Queries sum across all instances; can’t drill down to one host. Debugging difficult.	Always include instance label (pod name, hostname). Use by (instance) in queries. Drill down: which host is slow?

Interview Questions

Q — Alert if P99 latency exceeds 500ms. Build PromQL query. What metric type required?

— Requires Histogram metric (http_request_duration_seconds_bucket with le labels). Query: histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le)) > 0.5. Breaks down: rate(...[5m]) = req/sec per bucket. sum(...by(le)) = cumulative across instances. histogram_quantile(0.99, ...) = 99th percentile boundary. Result in seconds. Alert fires if >0.5s. Do NOT use Summary metric—they’re pre-calculated, unqueryable percentiles.

Q — Prometheus OOM-killing; cardinality explosion suspected. Diagnose and fix.

— Check cardinality: curl http://prometheus:9090/api/v1/label/__name__/values | wc -l (total unique metrics). Query dashboard: topk(10, count by (__name__) ({__name__=~".+"})) (largest metrics by series). Identify high-cardinality label: search source code for user_id, client_ip, request_id labels on counters. Fix: use metric_relabel_configs in Prometheus to drop that label before ingestion. Example: - source_labels: [user_id] action: drop. Enforce code review: never use unbounded labels. Result: cardinality drops; memory recovers.

Q — You want to alert if error rate ≥1% over last 5 minutes. Build PromQL rule.

— Query: (rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m])) > 0.01. Numerator: 5xx errors per second. Denominator: total requests per second. Ratio: error rate. >0.01 = ≥1%. Use rate() not irate()—stable over range, avoids micro-spike false positives. Alert fires if sustained ≥1% over 5 min window.