Tools / Monitoring and Observability Interview Questions
What is a dead man's switch alert and when should you use it?
A dead man's switch alert (also called a heartbeat alert or watchdog alert) is an alert that fires when it stops receiving a signal, rather than when it detects a problem. The pattern inverts the usual alerting logic: instead of "alert when metric X exceeds threshold Y", it says "alert if I have not heard from system X in the past N minutes."
The canonical use case is monitoring your monitoring system. If Prometheus crashes, it cannot emit metrics, so all your normal alerts go silent — and you would never know. A dead man's switch in an external system (Alertmanager's Watchdog alert, PagerDuty's dead man's switch feature, or a separate uptime monitor like Better Uptime or StatusCake) expects a regular "I'm alive" ping from your monitoring system every N minutes. If the ping stops, the external system fires an alert.
Other use cases:
- Scheduled batch jobs: Alert if the nightly ETL pipeline does not emit a completion metric within 2 hours of its scheduled start time.
- Queue consumers: Alert if a Kafka consumer stops consuming (no heartbeat emitted) — possibly indicating it is deadlocked or crashed without surfacing an error.
- Certificate renewal jobs: Ensure the cert-renewal cron job emits a success metric within 24 hours of its expected run time.
In Prometheus, the Alertmanager configuration ships a built-in Watchdog alert that fires continuously when healthy. Routing this alert to a dead man's switch service (Alertmanager's own Watchdog route, or a service like DeadMansSnitch) closes the loop.
More Related questions...