Grafana

TL;DR

• Grafana is a visualization and dashboarding layer, not a metrics store — it queries data sources (Prometheus, Loki, CloudWatch, etc.) rather than storing time-series itself.
• Dashboards should be designed around a specific question (incident triage vs. deep investigation); a 40-panel “everything” dashboard is slower to use than a focused one.
• Templated variables (e.g. $service) eliminate hand-copied per-service dashboards, but only affect panels whose queries explicitly reference the variable.
• Unified Alerting consolidates alert rule management across multiple data sources into Grafana itself, instead of relying solely on a single backend’s own alerting (e.g. Prometheus Alertmanager).
• Biggest gotcha: dashboards and alerts edited only through the UI aren’t reviewable, reproducible across environments, or auditable — the same governance gap Terraform/Jenkinsfiles solved for infrastructure applies here.

Core Concepts

Dashboard design principle

A dashboard should be designed around the specific question it answers, not maximize the amount of data shown. An incident-response dashboard benefits from a small number of high-signal panels — the golden signals (latency, traffic, error rate, saturation) — letting an on-call engineer orient in seconds. A deep-dive investigation dashboard, used deliberately after golden signals have narrowed down where to look, can reasonably carry more depth and panel count. Conflating both goals into one large “everything” dashboard serves neither well; it’s the failure mode that surfaces most clearly during a real incident, when someone is scanning frantically for the one panel that matters.

Templated variables

A templated variable (e.g. $service), driven by a query against available label values, turns one dashboard definition into something reusable across every service, with a dropdown to switch context, instead of maintaining dozens of nearly-identical hand-copied dashboards that drift out of sync the moment one is updated. The mechanism has a sharp edge: a variable only affects panels whose queries explicitly reference it. A panel with a hardcoded or variable-free query silently ignores the dashboard’s selected filter — switching the $service dropdown updates most panels but leaves one or two stubbornly showing all-services data.

Unified Alerting

Unified Alerting, built into Grafana itself, lets alert rules be defined and managed across multiple, different data sources — not just Prometheus, but Loki, CloudWatch, and others — in one consistent place, rather than being limited to whatever alerting capability a single backend natively supports. For an organization with metrics in Prometheus, logs in Loki, and cloud metrics in CloudWatch, this consolidates what would otherwise be separate alerting configuration per backend into a single rule-management surface inside Grafana.

Dashboards and alerts as code

UI-only edits to dashboards or alert rules aren’t reviewable via pull request, aren’t easily reproducible across environments (a staging Grafana instance silently diverging from production), and carry no audit trail of who changed what threshold and why. Provisioning dashboards and alert rules as version-controlled configuration files (JSON dashboard definitions, provisioning YAML) applies the same review-and-audit discipline already standard for Terraform, Kubernetes manifests, and CI pipeline definitions.

Grafana vs Prometheus (roles)

Dashboards-as-code vs UI-built

Command / Configuration Reference

# provisioning/dashboards/dashboards.yml — load dashboards from disk
apiVersion: 1
providers:
  - name: 'default'
    folder: 'Production'
    type: file
    options:
      path: /var/lib/grafana/dashboards

// Dashboard variable definition (templating.list entry)
{
  "name": "service",
  "type": "query",
  "query": "label_values(up, job)",
  "refresh": 2
}

// Panel query referencing the variable — required for the variable to take effect
{
  "expr": "rate(http_requests_total{job=\"$service\"}[5m])"
}

# provisioning/alerting/rules.yml — Unified Alerting rule as code
apiVersion: 1
groups:
  - orgId: 1
    name: api-golden-signals
    folder: Production
    rules:
      - title: High error rate
        condition: C
        for: 5m

Common Pitfalls

• Pitfall: A 40-panel dashboard is unusable during a real incident. Why: It optimizes for completeness rather than the specific question (“what’s broken right now”) an incident-response dashboard needs to answer. Fix: Build a focused golden-signals dashboard for triage; keep deep-dive panels on a separate investigation dashboard.
• Pitfall: Dozens of hand-copied per-service dashboards drift out of sync. Why: Each is maintained independently instead of sharing one templated definition. Fix: Use a dashboard variable (e.g. $service) driven by label values instead of duplicating dashboards.
• Pitfall: A dashboard variable dropdown changes most panels but one panel silently shows all-services data. Why: That panel’s query doesn’t reference the variable. Fix: Audit every panel query to confirm it uses $variable where filtering is expected.
• Pitfall: Alert thresholds change with no record of who changed them or why. Why: Alerts are edited only through the UI with no version control. Fix: Provision alert rules as code and require PR review for changes.
• Pitfall: Staging Grafana quietly diverges from production. Why: Dashboards were only ever edited in one environment through the UI. Fix: Provision dashboards from version-controlled files applied identically to every environment.

Interview Questions

• Design an incident-response dashboard for a customer-facing API. What goes on it, and what deliberately doesn’t? — Tests whether golden-signals-focused design is the instinct over maximizing information density.
• A dashboard variable dropdown changes most panels but one panel stubbornly shows all-services data. What’s wrong? — Tests whether the per-panel variable-reference requirement is understood.
• Why should dashboards and alerts be managed as code rather than purely through the Grafana UI? — Tests whether the review/audit/reproducibility argument is articulated clearly.
• What’s the architectural difference between Grafana and Prometheus, and why can’t Grafana replace Prometheus? — Tests understanding of visualization-layer vs. storage/query-engine separation of concerns.
• How does Unified Alerting change alert management for an organization using multiple observability backends? — Tests understanding of cross-data-source alert consolidation.