Prometheus

TL;DR

• Prometheus is a pull-based metrics system: it scrapes targets on an interval rather than receiving pushed data.
• PromQL is its query language; rate() on a counter is the single most important function to get right — raw counter values are cumulative totals, not rates.
• Exporters translate a system’s native stats into Prometheus’s exposition format, enabling monitoring of anything without modifying the monitored application.
• Alertmanager handles routing, grouping, deduplication, and inhibition of firing alerts — grouping/inhibition is what prevents an outage from becoming 200 individual pages.
• Biggest gotcha: alerting on fixed absolute thresholds (e.g. “CPU > 80%”) produces both false positives and false negatives because it ignores what’s normal for that specific service and workload.

Core Concepts

Pull-based scraping architecture

Prometheus scrapes (pulls) metrics from targets on a configured interval rather than targets pushing metrics to it. Prometheus itself controls scrape timing, so a failed scrape unambiguously means the target is unreachable — in a push model, “no data received” is ambiguous between “target is down” and “target had nothing to report.” Onboarding a new target only requires adding an entry to Prometheus’s scrape config; the target itself needs no reconfiguration to know where to push.

PromQL and rate()

A counter metric (e.g. http_requests_total) only ever increases, resetting to zero on process restart. Its raw value is a cumulative total since start, not a rate — reading it directly answers “how many requests total,” not “how many requests per second,” which is the question that actually matters operationally. rate(http_requests_total[5m]) computes the per-second average rate of increase over the window, correctly handling counter resets. Without rate(), a dashboard shows a meaningless ever-climbing line instead of an interpretable request rate. irate() is the instantaneous variant, using only the last two data points in the range — more responsive to spikes, noisier on volatile data. Gauges (values that go up and down, like memory usage) are read directly; rate() is meaningless on them.

Exporters

An exporter is a small process, typically run alongside the monitored system, that translates a native metrics format (a database’s internal stats, a hardware sensor reading, a legacy application’s log output) into Prometheus’s exposition format. This is what makes Prometheus viable in a heterogeneous production environment — legacy or third-party systems don’t need to be rewritten or modified; the exporter bridges the format gap. Common examples: node_exporter (host-level metrics), mysqld_exporter, blackbox_exporter (probing endpoints).

Alertmanager: routing, grouping, inhibition

Alertmanager receives alerts fired by Prometheus’s alerting rules and handles routing, grouping, silencing, and deduplication before notifications go out.

• Grouping combines related alerts (every instance of the same underlying problem) into a single notification instead of one per alert — the direct fix for “200 pages in 2 minutes during an outage.”
• Inhibition suppresses lower-priority alerts when a related higher-priority one is already firing — a cluster-down alert suppresses every individual pod-down alert within it, since those are downstream symptoms of the same root cause, not independent problems.
• Silences temporarily mute alerts matching a label set, typically during planned maintenance.

Alert design discipline

A fixed, absolute threshold (“CPU > 80%”) produces both false positives (80% CPU is normal for a batch job) and false negatives (80% may already be dangerous for a latency-sensitive API under a different workload) because it ignores what’s normal for that specific service at that specific time. Alerting on a symptom that directly reflects user impact — error rate, latency percentiles — or on deviation from a learned/contextual baseline is generally a more accurate signal than a raw resource metric compared against an arbitrary number. This maps to the “symptom-based over cause-based” alerting philosophy behind the four golden signals (latency, traffic, errors, saturation).

Prometheus pull vs push (remote_write)

Command / Configuration Reference

# prometheus.yml — minimal scrape config
scrape_configs:
  - job_name: 'node'
    scrape_interval: 15s
    static_configs:
      - targets: ['localhost:9100']

# Per-second request rate over 5m window
rate(http_requests_total[5m])

# Instantaneous rate (last two points only)
irate(http_requests_total[1m])

# Error ratio as a percentage
sum(rate(http_requests_total{status=~"5.."}[5m]))
  / sum(rate(http_requests_total[5m])) * 100

# 95th percentile latency from a histogram
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

# Alert: target down for 5 minutes
up == 0

# Aggregation across a label
sum by (job) (rate(http_requests_total[5m]))

# alertmanager.yml — grouping and inhibition
route:
  group_by: ['alertname', 'cluster']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['cluster', 'alertname']

Common Pitfalls

• Pitfall: Dashboard shows a raw counter value instead of rate(). Why: Counters are cumulative totals, not rates, so the raw value only ever climbs. Fix: Always wrap rate-of-change queries on counters in rate() or irate().
• Pitfall: On-call gets flooded with alerts during an outage. Why: No Alertmanager grouping or inhibition rules exist, so every individual alert fires as its own notification. Fix: Configure group_by and inhibit_rules so related alerts collapse into one notification.
• Pitfall: A fixed CPU threshold fires constantly on a normal batch workload while missing a real latency spike elsewhere. Why: Absolute thresholds don’t account for per-service, per-workload baselines. Fix: Alert on user-facing symptoms (error rate, latency) or dynamic baselines instead of static resource thresholds.
• Pitfall: Legacy systems go unmonitored indefinitely. Why: Teams assume Prometheus support requires rewriting the application. Fix: Check for an existing exporter before building custom instrumentation.
• Pitfall: Scrape target health itself isn’t monitored. Why: Teams assume “no metrics” is loud enough on its own. Fix: Alert on up == 0 explicitly so a silently failing scrape target is caught.

Interview Questions

• Explain exactly why rate() is necessary for a counter metric, and what goes wrong without it. — Tests genuine mechanical understanding versus copy-pasted PromQL usage.
• On-call received 200 pages in 2 minutes during a major outage. Walk through the Alertmanager fix. — Tests whether grouping and inhibition are the immediate, correct diagnosis.
• Why might alerting on latency/error rate be more effective than alerting on a raw CPU threshold? — Tests whether symptom-based versus cause-based alerting is understood.
• Compare Prometheus’s pull model to a push-based metrics system. When would push actually be preferable? — Tests whether the trade-off (ephemeral jobs, NAT/firewall constraints) is understood, not just the standard pull-model pitch.
• What problem do exporters solve, and how do they work mechanically? — Tests understanding of the exposition format and the separation between exporter and monitored system.