What is the Log Aggregation pattern and how does a centralised logging pipeline work?

The Log Aggregation pattern collects log output from every service instance and ships it to a centralised store where it can be searched, correlated, and analysed in one place. Without aggregation, diagnosing an incident across 50 service instances means SSHing into individual machines — impractical at scale.

A typical pipeline (EFK/ELK stack):

1. Emit structured logs — each service writes JSON-formatted log events to stdout (preferred in containers) or a log file. Structured logs include fields like timestamp, level, service, traceId, message.
2. Log shipper — Fluentd or Filebeat runs as a DaemonSet (one per node in Kubernetes) and tails container log files, applying parsing, filtering, and enrichment rules before forwarding.
3. Aggregator/processor — Logstash or Fluentd aggregator buffers, transforms (grok patterns, field extraction), and routes events.
4. Storage and search — Elasticsearch (or OpenSearch) indexes log events for full-text and structured queries.
5. Visualisation — Kibana (or OpenSearch Dashboards) provides dashboards, search, and alerting.

# Structured JSON log line emitted by a service
{
  "timestamp": "2026-04-22T09:01:23.456Z",
  "level":     "ERROR",
  "service":   "order-service",
  "traceId":   "4bf92f3577b34da6a3ce929d0e0e4736",
  "spanId":    "00f067aa0ba902b7",
  "message":   "Payment gateway timeout for order 42",
  "orderId":   42,
  "durationMs": 5001
}

The Correlation ID / Trace ID field is essential: it ties together all log lines from a single end-to-end request across every service that handled it, even when each service writes its logs to a different local file. A single Kibana query on traceId=4bf92f3... shows the entire request journey in chronological order.

Invest now in Acorns!!! 🚀
Join Acorns and get your $5 bonus!

Acorns is a micro-investing app that automatically invests your "spare change" from daily purchases into diversified, expert-built portfolios of ETFs. It is designed for beginners, allowing you to start investing with as little as $5. The service automates saving and investing. Disclosure: I may receive a referral bonus.

Invest now!!! Get Free equity stock (US, UK only)!

Use Robinhood app to invest in stocks. It is safe and secure. Use the Referral link to claim your free stock when you sign up!.

The Robinhood app makes it easy to trade stocks, crypto and more.

Webull! Receive free stock by signing up using the link: Webull signup.