OpenTelemetry & Distributed Tracing
OpenTelemetry & Distributed Tracing
TL;DR
- Distributed tracing visualizes single requests across microservices. Metrics show “is there a problem”; traces show “where”; logs show “what exactly.”
- Core concepts: Trace (full request journey), Span (unit of work), Trace ID (passed between services), Context Propagation (headers or message attributes).
- OTel Collector receives data from services via Receivers, processes it (batch, sample, scrub PII), exports to backends (Jaeger, Prometheus, Loki, etc.).
- Head-based sampling (decide at start) is cheap but misses errors that happen mid-trace. Tail-based sampling (decide at collector) catches all errors but needs memory.
- Context propagation breaks if custom HTTP clients don’t use auto-instrumentation. Result: “broken traces” starting fresh at each service.
- Use structured trace IDs to search across logs, metrics, and traces—single trace_id correlates everything.
Three Pillars of Observability
| Pillar | Question | Tool | Data Type | Use |
|---|---|---|---|---|
| Metrics | Is there a problem? | Prometheus, Grafana | Time-series counters, gauges, histograms | Alerts, dashboards, trend analysis |
| Traces | Where is the problem? | OpenTelemetry, Jaeger | Waterfall spans, latency, dependencies | Debugging request flow, root cause |
| Logs | What exactly is the problem? | Elasticsearch, Loki | Structured events, error messages, stack traces | Deep debugging, audit trail |
Core Concepts
Trace
Complete journey of a single request through the distributed system. Globally unique trace_id. Represents entire request lifecycle across all services.
Span
Single unit of work within a trace. Examples: HTTP request, database query, cache lookup, gRPC call. Has span_id, parent_span_id, start_time, end_time, attributes.
Context Propagation
Mechanism to pass trace_id from Service A to Service B so spans connect. HTTP: inject into headers (W3C traceparent standard). Async: inject into message attributes (SQS, Kafka, RabbitMQ).
OTel Collector
Central component that receives, processes, and exports telemetry. Decouples services from backend choice. Can run as DaemonSet (Kubernetes) or sidecar.
Sampling
Decision to keep or drop trace spans to manage storage costs and performance. Head-based (early decision) vs. tail-based (late, smarter decision).
OpenTelemetry Collector Pipeline
SERVICES (generate spans)
↓
RECEIVERS (OTLP, Jaeger, Zipkin, etc.)
↓
PROCESSORS (batch, sample, scrub PII, error detection)
↓
EXPORTERS (Jaeger, Prometheus, Loki, Splunk, etc.)
↓
BACKENDS (UI for visualization, storage)Sampling Strategies Comparison
| Strategy | Decision Point | CPU Cost | Storage | Error Capture | Use Case |
|---|---|---|---|---|---|
| Head-based (Random 10%) | Start of request | Low | 10x baseline | May miss errors | High-throughput systems |
| Head-based (Error-aware) | Start; flag errors | Low–Med | 15–50x | All errors | Balanced approach |
| Tail-based (Error + Anomaly) | After span completion | High | 20–100x | 100% errors + anomalies | Critical systems, debugging |
| Tail-based (All traces) | After span completion | Very high | Unlimited | 100% traces | Full observability (expensive) |
Context Propagation Patterns
| Transport | Mechanism | Header/Attribute | Example |
|---|---|---|---|
| HTTP | Inject headers | traceparent (W3C standard) |
traceparent: 00-[trace_id]-[span_id]-[flags] |
| gRPC | Inject metadata | traceparent or custom metadata |
Auto-instrumented by OTel SDK |
| SQS | Message Attributes | Custom key-value pair | Extract in Python, inject into Message Attributes, extract in Go consumer |
| Kafka | Message headers | Custom headers | Similar to SQS; OTel provides helpers |
| Database | Query comments | /*trace_id=[value]*/ |
Helps correlate slow queries with traces |
Configuration Reference: OTel Collector Config
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
jaeger:
protocols:
grpc:
endpoint: 0.0.0.0:14250
thrift_http:
endpoint: 0.0.0.0:14268
processors:
batch:
send_batch_size: 1000
timeout: 10s
memory_limiter:
check_interval: 1s
limit_mib: 2048
attributes:
actions:
- key: service.version
value: "v1.2.3"
action: insert
tail_sampling:
policies:
- name: error-traces
type: status_code
status_code:
status_codes: [ERROR]
- name: slow-traces
type: latency
latency:
threshold_ms: 1000
exporters:
jaeger:
endpoint: jaeger-collector:14250
prometheus:
endpoint: "0.0.0.0:8889"
otlp:
endpoint: loki-distributor:4317
service:
pipelines:
traces:
receivers: [otlp, jaeger]
processors: [memory_limiter, attributes, tail_sampling, batch]
exporters: [jaeger]
metrics:
receivers: [otlp]
processors: [batch]
exporters: [prometheus]
logs:
receivers: [otlp]
processors: [batch]
exporters: [otlp]Common Pitfalls
| Pitfall | Why It Happens | Fix |
|---|---|---|
| Broken trace chain | Custom HTTP client not using OTel auto-instrumentation. W3C traceparent header not injected. |
Use OTel SDKs for all HTTP clients. Enforce code review: check for inject() calls. Add tests verifying header propagation. |
| Async queue loses context | SQS/Kafka message doesn’t carry traceparent. Each consumer starts new trace. |
Extract trace_id in producer, inject into message attributes. Extract in consumer, resume trace context. Use OTel helpers. |
| Trace cardinality explosion | High-cardinality attributes (user IDs, request IDs) create millions of unique span combinations. Collector memory exhausted. | Don’t add unbounded attributes to spans. Use sampling. Or: strip high-cardinality fields in Processor before export. |
| Tail-based sampling memory leak | Collector buffers spans waiting for trace completion. Large requests or slow services hold memory indefinitely. | Set memory limits in collector config. Configure timeout in processors. Alert on collector memory usage. Cap span buffering. |
| No context in error logs | Application logs error without trace_id. Impossible to correlate with trace. |
Inject trace_id into all structured logs. Use OTel context propagation in logger. Example: logger.info("error", trace_id=span.trace_id). |
| Over-sampling discards errors | Head-based sampling at 1% probability. Rare error paths get dropped. Errors invisible in tracing backend. | Use head-based + error-aware sampling: always sample errors. Or: tail-based sampling to catch errors 100%. Accept higher cost. |
Interview Questions
Q — Checkout request fails with generic 500 error. Logs show only checkout-service. How does tracing find root cause?
— Without traces: manually search logs of inventory, payment, shipping services at approximate timestamp. Slow, error-prone. With traces: search Jaeger for trace_id of failed request. UI displays waterfall: every service, every span, latencies, errors. Visual inspection shows payment-service Span took 5s and returned 500. Root cause: payment API timeout. Traces compress hours of log searching into seconds. Single trace_id correlates all services, all logs, all metrics.
Q — Python app publishes SQS message to Go consumer. Preserve trace context across queue. How?
— HTTP headers don’t work for queues. Solution: In Python producer, extract traceparent from current span context. Inject into SQS Message Attributes (key-value map). Publish message. In Go consumer, poll SQS. Extract traceparent from Message Attributes. Use OTel SDK to resume trace context from that traceparent. Start child span for processing. Result: trace chain unbroken; waterfall shows Python → SQS → Go.
Q — You’re deciding: head-based 10% sampling vs. tail-based error-aware sampling. Trade-offs?
— Head-based (10%): cheap (low CPU on services), low storage. Downside: 90% of errors discarded—invisible in Jaeger. If rare error path, you’ll never see it. Tail-based (error-aware): higher CPU (collector buffers all spans), higher storage (100% of errors, anomalies kept). Upside: all errors visible, debuggable. Best practice for production: tail-based error sampling. Cost is worth it. If budget tight: head-based + explicit flagging (mark errors for 100% sampling regardless of random sample rate).