Logging

TL;DR

• Logging backends fall into two indexing models: full-text (Elasticsearch/OpenSearch) and metadata-label-only (Loki) — the choice is a permanent trade-off between search power and operational cost.
• Structuring logs at collection time (Fluent Bit) shifts parsing cost from every query to once, at ingestion.
• Log correlation via a shared trace/request ID is what makes debugging a multi-service request tractable.
• “Log everything at DEBUG in production” is a cost problem and a signal-to-noise problem, not a free safety net.
• Biggest trade-off: full-text indexing buys powerful free-text search at real storage/compute cost that requires active lifecycle management to stay bounded.

Core Concepts

Indexing models: Loki vs. full-text

Loki indexes only metadata labels (the same model Prometheus uses for metrics), not log line content. Queries filter by label first (fast, indexed), then grep the matching log content directly (not pre-indexed). This is cheap to store and operate at scale but slower for broad free-text search across large volumes. OpenSearch/Elasticsearch fully index log content itself, enabling rich, fast free-text search — at the cost of significantly higher storage and compute per byte ingested.

Structuring at collection time

Fluent Bit and similar log shippers/processors extract fields (status_code, duration_ms) from raw log lines during ingestion rather than leaving them as unstructured text to parse at query time. Every downstream query, dashboard, and alert can then filter/aggregate on those fields directly. Query-time regex parsing re-runs the same extraction cost on every single search instead of paying it once during ingestion.

Log correlation

Without a shared trace/request ID propagated across every service a request touches, reconstructing a distributed request’s behavior means manually correlating logs across separate streams by timestamp proximity — slow and error-prone. A propagated correlation ID (ideally the same trace ID OpenTelemetry generates) turns that into a single query filtered by one identifier, pulling every relevant log line from every service in one shot.

Index lifecycle management

Full-text indexing’s storage/compute cost requires active management at scale: rollover policies (create a new index at a size/time threshold), retention windows (delete or archive indices past a certain age), and tiered/cold storage for older indices. Without this, logging backend cost grows unbounded as volume grows.

Log volume and level discipline

Excessive log volume increases storage cost and ingestion/indexing load, and — separately — makes the relevant log line during an incident harder to find, buried in noise. Deliberate log-level and content discipline (log what’s diagnostically useful, at an appropriate level) improves both cost and incident-response speed.

Logs vs. Metrics vs. Traces

Command / Configuration Reference

# fluent-bit.conf equivalent (YAML) — parse and structure at collection time
pipeline:
  inputs:
    - name: tail
      path: /var/log/app/*.log
  filters:
    - name: parser              # extracts structured fields from raw log lines
      match: app.*
      key_name: log
      parser: json
  outputs:
    - name: loki                # ships structured logs to Loki with labels
      match: app.*
      labels: job=app, env=prod

# LogQL (Loki query language) — label filter first, then content grep
{job="app", env="prod"} |= "error" | json | status_code >= 500

// Elasticsearch/OpenSearch Index Lifecycle Management (ILM) policy
{
  "policy": {
    "phases": {
      "hot":    { "actions": { "rollover": { "max_size": "50gb", "max_age": "1d" } } },
      "warm":   { "min_age": "7d", "actions": { "shrink": { "number_of_shards": 1 } } },
      "delete": { "min_age": "30d", "actions": { "delete": {} } }
    }
  }
}

# Propagate a correlation/trace ID through a request (shell example)
curl -H "traceparent: 00-$(uuidgen)-0000000000000001-01" https://service-b/api

Common Pitfalls

• Pitfall: Migrating from Elasticsearch to Loki and hitting slow free-text search. Why: Loki indexes labels only, not log content, so unindexed content search is inherently slower. Fix: Understand the indexing trade-off before migrating; structure logs with rich labels to compensate.
• Pitfall: Logging backend storage cost grows quietly for months. Why: No index lifecycle management (rollover/retention/tiering) configured. Fix: Set explicit ILM policies before volume grows unchecked.
• Pitfall: Debugging a distributed request takes hours of manual timestamp correlation. Why: No shared correlation ID propagated across services. Fix: Propagate a trace/request ID (ideally shared with the tracing system) through every service boundary.
• Pitfall: Production logs left at DEBUG level everywhere. Why: Treated as a free safety net with no downside. Fix: Apply deliberate log-level and content discipline; log what’s diagnostically useful, not everything.
• Pitfall: Parsing log content at query time via regex. Why: Re-runs parsing cost on every query instead of once at ingestion. Fix: Structure and extract fields at collection time (Fluent Bit or equivalent).

Interview Questions

• A team wants to migrate from Elasticsearch to Loki for cost reasons. What trade-off do you make sure they understand first? — Tests whether the indexing-model difference and its query-performance consequence is understood.
• A distributed request across 5 services fails, and debugging it means manually correlating 5 separate log streams by timestamp. What’s missing, and how do you fix it? — Tests whether correlation ID propagation is the immediate diagnosis.
• Why is “log everything at DEBUG in production, just in case” a bad practice beyond storage cost? — Tests whether the incident-response signal-to-noise argument is articulated, not just the cost argument.
• When would you choose full-text indexing over label-based indexing for a logging backend? — Tests understanding of the search-power-versus-cost trade-off.
• How would you design log structure to make future queries and alerts efficient? — Tests understanding of collection-time structuring versus query-time parsing.