How does observability apply to event-driven and asynchronous architectures?

Observability in event-driven architectures (EDA) — systems built around message queues like Apache Kafka, RabbitMQ, or AWS SQS — presents distinct challenges because requests do not follow a synchronous request-response path. A single business transaction might produce events consumed by multiple services asynchronously, making traditional HTTP-trace-based observability incomplete.

Message tracing: The core technique is propagating trace context through message headers. Just as HTTP requests carry traceparent headers, Kafka messages carry trace context in their headers map. When a consumer reads a message and creates a child span, it extracts the producer's trace ID from the message headers. OpenTelemetry's Kafka instrumentation handles this automatically, enabling end-to-end traces that span the Kafka boundary.

Consumer lag monitoring: In Kafka, consumer lag (the difference between the latest offset and the consumer group's committed offset) is the primary signal of throughput problems. A growing lag means the consumer is falling behind producers. Kafka's JMX metrics and the kafka_consumer_group_lag metric (exported by the Kafka Exporter for Prometheus) are essential.

Message queue depth: In SQS or RabbitMQ, queue depth (number of messages waiting) and message age (oldest message waiting time) signal consumer health and backpressure.

Poison pill detection: Messages that consistently fail processing and end up in dead-letter queues (DLQs) must be monitored. A growing DLQ count with no alert is a silent data loss scenario.