Tracing

TL;DR

• Distributed tracing shows the full journey of a single request across multiple services as one connected timeline, answering “where did the time go” — something logs and metrics alone cannot.
• Trace context propagation (passing trace/span IDs in request headers) is required; without it, spans from different services cannot be linked into a single trace.
• Tempo (Grafana-integrated, efficient at scale) and Jaeger (CNCF standard) are the dominant tracing backends; both depend on the same propagation mechanism.
• Gaps between parent and child spans typically indicate uninstrumented application code (serialization, business logic, locks); gaps are signals for deeper instrumentation, not tool malfunction.
• Tail-based sampling (retain/discard decisions made after seeing the full trace) guarantees error-containing traces are captured, unlike head-based sampling (decided at trace start).
• Biggest gotcha: dismissing a trace as “not useful” when the gap itself is the signal for exactly where deeper instrumentation is needed.

Core Concepts

Logs vs. metrics vs. tracing: different questions

Metrics answer “what’s the system’s aggregate health?” — latency percentiles, error rates, throughput. Logs capture discrete events per service. Neither reconstructs the actual journey of one specific request across multiple services. Distributed tracing answers “for this one request, where did the time go, and in what order?” — it shows the full request path as one connected timeline, pinpointing exactly which hop added latency rather than just confirming each service “looked fine” independently.

Trace structure: spans and trace context

A trace is a tree of spans, each representing one operation (incoming request, database query, downstream service call) with a start time and duration. The trace shows the hierarchy: a parent span encompasses child spans, and gaps between them indicate uninstrumented code paths. Trace context propagation — passing a trace ID and parent span ID across service boundaries in HTTP headers or message metadata — is the mechanism that links spans from different services into one trace. Without propagation, spans are disconnected timing data with no way to assemble a connected timeline; propagation is a structural requirement, not optional.

Tempo and Jaeger

Tempo is a Grafana-integrated tracing backend, architected to operate cost-effectively at scale using object storage (S3, GCS) for span storage rather than indexing. Jaeger is the earlier, CNCF-standardized tracing system, with a more traditional storage model (Elasticsearch, Cassandra). Both are functionally similar for the core use case — capturing and querying traces — but differ in operational cost profile and ecosystem integration. Both depend entirely on trace context propagation.

Reading traces: spans and gaps

A gap between a parent span and its child spans — where child spans account for only a fraction of the parent’s duration — indicates time spent in application code outside any instrumented operation: serialization, business logic, waiting on locks, or calls to unin instrumented dependencies. This gap is not a tracing tool malfunction; it’s a precise signal of exactly where deeper instrumentation should be added. Teams that correctly interpret gaps as instrumentation hints can quickly identify blind spots in observability.

Sampling strategy: head-based vs. tail-based

Traces are data-heavy per unit captured; a single trace with many spans across many services is substantially larger than a typical metric or log line. At high request volume, capturing every trace is prohibitively expensive. Head-based sampling decides at trace start (before the outcome is known) which traces to sample, necessarily using heuristics like “sample every Nth request.” Tail-based sampling makes the retain/discard decision after seeing the full trace, allowing rules like “keep every trace containing an error.” Tail-based sampling is more processing-intensive but guarantees that error-containing traces are retained, making it a worthwhile trade-off for incident investigation where missing the exact failing trace is costly.

Correlation IDs and tracing

A correlation ID (the same term applied to logs) is another key: the same ID should flow through request headers and into logs and tracing, allowing a team to pivot directly from a trace of interest into the detailed logs for a specific span, creating a connected observability workflow.

Tempo vs. Jaeger

Configuration / Command Reference

# Tempo configuration example (distributed deployment via Helm/Kubernetes)
# Key concept: pipeline stages — ingest → batching → storage

# Docker Compose quick-start
docker run -d -p 3200:3200 -p 4317:4317 grafana/tempo:latest

# Jaeger quick-start (all-in-one)
docker run -d -p 16686:16686 -p 4317:4317 jaegertracing/all-in-one:latest

# Send trace data via OpenTelemetry Protocol (OTLP) to Tempo or Jaeger
curl -X POST http://localhost:4317/v1/traces \
  -H "Content-Type: application/json" \
  -d '{"resourceSpans": [...]}'

# Query traces via Tempo's API (example: search by tag)
curl "http://tempo:3200/api/search?tags=service.name%3Dmy-service"

# Query traces via Jaeger UI
# Navigate to http://localhost:16686

Common Pitfalls

• Pitfall: Trace context propagation missing across one asynchronous messaging hop, silently breaking the connected trace at that boundary. Why: Propagation is required across every service boundary, not just synchronous HTTP calls; async patterns (message queues, pub/sub) are easily overlooked. Fix: Audit every service-to-service boundary; ensure trace ID is explicitly extracted from and written to message headers.
• Pitfall: A large, unexplained gap between parent and child spans dismissed as a tracing tool issue. Why: Teams unfamiliar with trace reading may assume gaps indicate malfunction rather than uninstrumented code paths. Fix: Treat gaps as instrumentation signals; identify and instrument the code path the gap points to.
• Pitfall: Uniform low-rate head-based sampling missing the specific trace containing an error during a real production incident. Why: Head-based sampling decides before the outcome is known; a uniformly low rate may simply miss the trace containing the failure being investigated. Fix: Implement tail-based sampling to guarantee error-containing traces are retained.
• Pitfall: Tracing backend storage cost growing unexpectedly at scale due to 100% sampling left as default. Why: Without a deliberate sampling strategy, every trace is persisted, and data volume grows linearly with request volume. Fix: Choose a sampling strategy based on request volume and incident-investigation needs; tune over time.

Interview Questions

• A request touching six services is intermittently slow, and each service’s logs look fine individually. Walk through how tracing helps diagnose this. — Tests whether the relative-timing and single-connected-timeline value of tracing is understood versus logs.
• A trace shows a large gap between a parent span and its child spans. What does that mean, and what should be done about it? — Tests whether gaps are interpreted as instrumentation signals, not tool malfunctions.
• Why would you choose tail-based sampling over head-based sampling for a production service, despite added processing cost? — Tests understanding of tail-based sampling’s error-guarantee value as a deliberate trade-off.
• How would you ensure trace context propagation across both synchronous and asynchronous service boundaries? — Tests awareness that propagation is not limited to HTTP, and must be configured explicitly in message-based patterns.