What is the difference between an alert and a notification in observability?

The terms "alert" and "notification" are often used interchangeably but represent different stages in the incident response pipeline. Understanding the distinction helps design more effective on-call systems.

An alert is the detection event itself — the result of evaluating a rule against metric or log data and finding that a condition is satisfied. In Prometheus, an alerting rule defines a PromQL expression and a duration threshold. When the expression is continuously true for the specified duration (e.g., 5 minutes), Prometheus changes the alert state from inactive to pending to firing. The alert is an internal state within the monitoring system.

A notification is how the alert is communicated to a human or another system. Alertmanager receives firing alerts from Prometheus, applies grouping, inhibition, and silencing, and then routes them to receivers — Slack channel, PagerDuty incident, email, or webhook. The notification is the downstream artifact of the alert.

This two-stage architecture is important because it allows sophisticated routing: the same alert can send a low-severity Slack message during business hours and a PagerDuty page at night. Alerts can be silenced during maintenance windows (suppressing notifications) without disabling the alerting rule. Multiple alerts can be grouped into a single notification to reduce noise.

Alertmanager also handles deduplication: if Prometheus sends the same alert 100 times (once per evaluation cycle), Alertmanager fires only one notification and re-notifies only after a configured repeat interval or when the alert recovers.