OpenTelemetry

TL;DR

• OpenTelemetry (OTel) is a vendor-neutral standard for instrumentation APIs, SDKs, and a Collector — it decouples “how telemetry is generated” from “where it’s sent.”
• Covers three signal types: traces, metrics, logs — instrumented once, exported anywhere via configuration.
• Auto-instrumentation gives zero-code-change baseline spans; manual instrumentation adds business context auto-instrumentation structurally cannot know.
• Semantic conventions standardize attribute names (http.response.status_code, db.system) so telemetry from different teams/services is queryable together.
• The Collector centralizes batching, retries, routing, and sampling outside application processes.
• Biggest trade-off: sampling strategy is mandatory at scale — 100% capture is rarely affordable, and the wrong sampling choice silently drops the trace that mattered.

Core Concepts

SDKs and the vendor-neutral API

SDKs are the language-specific implementation of OTel’s API. Applications instrument against this common API; the destination backend is exporter configuration, not application code. This directly solves the lock-in problem created by instrumenting against a proprietary APM SDK — under that model, evaluating a new vendor requires re-instrumenting every service. Under OTel, switching or adding a backend is a config change.

Instrumentation: auto vs. manual

Auto-instrumentation hooks into common frameworks/libraries (HTTP servers, DB drivers) via bytecode manipulation, monkey-patching, or language-specific hooks to produce standard spans with no application code changes. Manual instrumentation adds spans, attributes, and events deliberately in application code — this is the only way to capture business-specific context (an order ID, a feature-flag state, a branch in business logic) that auto-instrumentation has no visibility into. Production systems typically combine both: auto-instrumentation for broad baseline coverage, manual instrumentation layered on top for the context that actually explains an incident.

Semantic conventions

Standardized, agreed-upon attribute names (http.response.status_code, db.system, service.name) so telemetry from different services and teams can be correlated and queried consistently. Without them, semantically identical data under different names (http.status vs. httpStatusCode) becomes unqueryable together in a shared dashboard or alert — defeating a core purpose of standardization.

The Collector

A separate, deployable process that centralizes telemetry-handling concerns outside individual application processes: batching, retry logic, backend routing, protocol translation (OTLP to vendor-specific formats), and sampling decisions. Applications export simply to the Collector; the Collector owns backend failover, filtering, and routing centrally instead of duplicating that logic in every application’s SDK config. Grafana Alloy is a common OTel Collector implementation. Deployment patterns: agent (sidecar/daemonset, one per host) and gateway (centralized, receives from agents) — often used together.

Sampling strategy

Capturing 100% of traces at real request volume creates real network, storage, and backend-processing cost that often exceeds the marginal value of the extra traces. Options:

• Head-based sampling — decision made at trace start, cheap, no visibility into how the trace turns out.
• Tail-based sampling — decision made after the full trace completes, can specifically retain all error traces, more resource- and memory-intensive (the Collector must buffer the whole trace).
• Error-biased / hybrid — low uniform rate for successful traces, 100% retention for errors.

The mistake is not choosing deliberately — either exporting everything (unsustainable cost) or sampling too aggressively by default (missing the rare trace that mattered).

Signal Types Comparison

Auto vs. Manual Instrumentation

Command / Configuration Reference

# otel-collector-config.yaml — minimal Collector pipeline
receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch: {}                     # batches telemetry before export, reduces network overhead
  tail_sampling:                 # tail-based sampling processor
    policies:
      - name: errors-policy
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: probabilistic-policy
        type: probabilistic
        probabilistic: {sampling_percentage: 10}

exporters:
  otlp:
    endpoint: "backend.example.com:4317"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, tail_sampling]
      exporters: [otlp]

# Auto-instrumentation via Java agent — zero code changes
java -javaagent:opentelemetry-javaagent.jar -jar myapp.jar

# Set OTLP exporter endpoint via environment (language-agnostic)
export OTEL_EXPORTER_OTLP_ENDPOINT="http://otel-collector:4317"
export OTEL_SERVICE_NAME="checkout-service"

# Manual span creation (Python)
from opentelemetry import trace
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("process-order") as span:
    span.set_attribute("order.id", order_id)   # business context auto-instrumentation can't provide

Common Pitfalls

• Pitfall: Instrumenting directly against a commercial APM vendor’s proprietary SDK. Why: Couples application code to a specific backend. Fix: Instrument against the OTel API; configure the backend via exporters only.
• Pitfall: Inconsistent attribute naming across teams (http.status vs. httpStatusCode). Why: No enforced semantic conventions. Fix: Standardize on OTel semantic conventions and lint for them in code review or CI.
• Pitfall: Relying solely on auto-instrumentation. Why: It cannot capture business-specific context by construction. Fix: Layer manual spans/attributes for business-critical paths.
• Pitfall: Defaulting to 100% trace sampling in production. Why: Cost and processing overhead scale linearly with volume and are rarely budgeted for. Fix: Choose a deliberate sampling strategy (head-based, tail-based, or hybrid) matched to volume and investigation needs.
• Pitfall: Exporting directly from every application to the backend instead of through a Collector. Why: Duplicates batching/retry/routing logic per service and complicates backend migration. Fix: Route all telemetry through a Collector.

Interview Questions

• Explain how OpenTelemetry prevents vendor lock-in, mechanically. — Tests whether the API-versus-backend decoupling is understood, not just cited as a benefit.
• Two teams’ services use inconsistent attribute names for the same concept. Why does this matter, and how do you fix it? — Tests understanding of semantic conventions as a cross-team correlation requirement.
• Design a sampling strategy for a high-volume service where every error trace must be captured. — Tests whether tail-based or error-biased sampling is the reasoned answer, not a default guess.
• Why does the Collector exist as a separate component instead of exporting directly from applications? — Tests understanding of centralized processing versus per-application duplication.
• What’s the trade-off between auto- and manual instrumentation? — Tests whether the business-context gap in auto-instrumentation is understood.