What is the Log Aggregation pattern and how does a centralised logging pipeline work?

The Log Aggregation pattern collects log output from every service instance and ships it to a centralised store where it can be searched, correlated, and analysed in one place. Without aggregation, diagnosing an incident across 50 service instances means SSHing into individual machines — impractical at scale.

A typical pipeline (EFK/ELK stack):

1. Emit structured logs — each service writes JSON-formatted log events to stdout (preferred in containers) or a log file. Structured logs include fields like timestamp, level, service, traceId, message.
2. Log shipper — Fluentd or Filebeat runs as a DaemonSet (one per node in Kubernetes) and tails container log files, applying parsing, filtering, and enrichment rules before forwarding.
3. Aggregator/processor — Logstash or Fluentd aggregator buffers, transforms (grok patterns, field extraction), and routes events.
4. Storage and search — Elasticsearch (or OpenSearch) indexes log events for full-text and structured queries.
5. Visualisation — Kibana (or OpenSearch Dashboards) provides dashboards, search, and alerting.

# Structured JSON log line emitted by a service
{
  "timestamp": "2026-04-22T09:01:23.456Z",
  "level":     "ERROR",
  "service":   "order-service",
  "traceId":   "4bf92f3577b34da6a3ce929d0e0e4736",
  "spanId":    "00f067aa0ba902b7",
  "message":   "Payment gateway timeout for order 42",
  "orderId":   42,
  "durationMs": 5001
}

The Correlation ID / Trace ID field is essential: it ties together all log lines from a single end-to-end request across every service that handled it, even when each service writes its logs to a different local file. A single Kibana query on traceId=4bf92f3... shows the entire request journey in chronological order.