Grafana Alloy

TL;DR

• Alloy consolidates metrics, logs, and traces collection into one unified agent with one configuration model, replacing the traditional multi-agent pattern (separate agents per telemetry type).
• Pipelines — receiver → processors → exporter — are the core composable configuration concept, enabling consistent handling of all telemetry types through a single declarative pattern.
• OTLP (OpenTelemetry Protocol) support decouples applications from any specific observability backend; switching backends doesn’t require re-instrumentation.
• High-cardinality label filtering at the collector level (before backend storage) protects metrics database performance while preserving high-cardinality data in logs/traces where it’s less structurally damaging.
• Centralized pipeline configuration enforces consistent labeling, sampling rates, and cost controls across teams, avoiding fragmentation when teams independently configure collection.
• Biggest gotcha: running multiple single-purpose agents in parallel for years without evaluating the operational consolidation gains of a unified collector.

Core Concepts

Multi-agent overhead vs. unified collection

Historically, observability requires a metrics agent, a separate log shipper, and a separate trace collector — each with distinct configuration syntax, resource footprint, upgrade cycle, and failure modes. Multiplied across every host in a fleet, this compounds operational overhead: three agents to deploy, three to upgrade, three failure modes to debug independently. A unified collector consolidates this to one agent, one configuration model, one footprint per host.

Pipelines: receiver → processor → exporter

Alloy’s core abstraction is a pipeline — a declarative flow of components: a receiver that ingests telemetry (from an application via OTLP, from a file, from a scrape target), optional processors that transform/filter/enrich data in transit, and an exporter that sends the result to a backend (Prometheus, Loki, Tempo, or third-party destinations). This composable model means the same configuration pattern handles metrics, logs, and traces, rather than learning three separate agents’ entirely different languages and operational quirks.

OTLP: vendor-neutral instrumentation

OTLP (OpenTelemetry Protocol) is a standardized, vendor-neutral wire format for telemetry data. Instrumenting an application to emit OTLP once means the collector can route that data to any OTLP-compatible backend. This decouples application instrumentation from any specific vendor’s proprietary format — switching or adding an observability backend later requires no application re-instrumentation, a significant operational advantage for long-term flexibility.

High-cardinality label filtering

A common, deliberate practice: dropping or aggregating away high-cardinality labels (a raw user ID, a full request path with dynamic segments) before metrics reach a time-series backend. High cardinality is a severe performance and storage problem for metrics systems — a unique label value per user multiplies the number of distinct time series the backend must track. Filtering at the collector level protects backend performance; that same high-cardinality data can still exist safely in logs or traces, where high cardinality is structurally less damaging.

Centralized configuration management

A platform team defining and distributing pipeline configuration centrally — rather than letting every team independently configure their own collection — enforces consistency: uniform labeling conventions, consistent sampling rates, and organization-wide cost/performance management. Without centralization, every team’s independently-chosen conventions fragment the data shape, making cross-team incident correlation and organization-wide cost analysis significantly harder.

Single-purpose agents vs. Alloy (unified collector)

Configuration / Command Reference

# Example Alloy pipeline: ingest Prometheus metrics, filter high-cardinality, export to Prometheus
prometheus.scrape "http_server" {
  targets = [{"__address__" = "localhost:8080"}]
  forward_to = [prometheus.relabel.filter.receiver]
}

prometheus.relabel "filter" {
  rule {
    # Drop high-cardinality user_id label before export
    action = "labeldrop"
    regex = "user_id"
  }
  forward_to = [prometheus.remote_write.example.receiver]
}

prometheus.remote_write "example" {
  endpoint {
    url = "http://prometheus:9090/api/v1/write"
  }
}

# Log collector pipeline: read from file, enrich with attributes, export to Loki
loki.source.file "app_logs" {
  targets = [{__path__ = "/var/log/app/*.log"}]
  forward_to = [loki.process.enrich.receiver]
}

loki.process "enrich" {
  stage.json {
    expressions = {timestamp = "ts", message = "msg", level = "level"}
  }
  forward_to = [loki.write.example.receiver]
}

loki.write "example" {
  loki_push_api {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

Common Pitfalls

• Pitfall: Running separate metrics, logs, and traces agents on every host for years without evaluating unified collection. Why: The operational overhead (three deployments, three upgrades, three failure modes) compounds across the fleet, but stays invisible until consolidation is explicitly considered. Fix: Evaluate Alloy or similar unified collectors; the operational win at fleet scale justifies migration effort.
• Pitfall: Applications instrumented directly against a vendor-specific telemetry format. Why: When an organization later wants to switch or add observability backends, every instrumented service requires code changes and re-deployment. Fix: Instrument applications to emit OTLP; ensure collectors translate vendor formats to OTLP as needed.
• Pitfall: High-cardinality labels (raw user ID, dynamic request paths) reaching a Prometheus-compatible backend unfiltered. Why: Each unique label value creates a distinct time series; high cardinality causes severe storage bloat and query performance degradation in metrics databases. Fix: Configure collector processors to drop or aggregate high-cardinality labels before export to metrics backends; preserve them in logs/traces.
• Pitfall: Every team independently configuring telemetry collection standards (labeling conventions, sampling rates). Why: Fragmented configurations make cross-team incident correlation slow and error-prone; organization-wide cost analysis becomes unreliable. Fix: Centralize pipeline configuration definitions; distribute them to teams as templates or required standards.

Interview Questions

• What operational problem does a unified telemetry collector like Alloy solve beyond reducing agent count? — Tests whether the consolidated configuration, upgrade, and resource-footprint arguments are articulated precisely.
• Why does OTLP support matter for an organization that might switch observability backends someday? — Tests understanding of vendor-neutral instrumentation as a long-term flexibility benefit.
• A metrics backend is experiencing severe storage and query performance issues. What collector-level cause should be checked first? — Tests whether high-cardinality label filtering is immediately recognized as a common, severe culprit.
• How would you design a centralized pipeline configuration approach for a multi-team organization? — Tests understanding of consistency benefits (labeling, sampling) and enforcement mechanisms.