Tools / Monitoring and Observability Interview Questions
What is a dead man's switch alert and when should you use it?
A dead man's switch alert (also called a heartbeat alert or watchdog alert) is an alert that fires when it stops receiving a signal, rather than when it detects a problem. The pattern inverts the usual alerting logic: instead of "alert when metric X exceeds threshold Y", it says "alert if I have not heard from system X in the past N minutes."
The canonical use case is monitoring your monitoring system. If Prometheus crashes, it cannot emit metrics, so all your normal alerts go silent — and you would never know. A dead man's switch in an external system (Alertmanager's Watchdog alert, PagerDuty's dead man's switch feature, or a separate uptime monitor like Better Uptime or StatusCake) expects a regular "I'm alive" ping from your monitoring system every N minutes. If the ping stops, the external system fires an alert.
Other use cases:
- Scheduled batch jobs: Alert if the nightly ETL pipeline does not emit a completion metric within 2 hours of its scheduled start time.
- Queue consumers: Alert if a Kafka consumer stops consuming (no heartbeat emitted) — possibly indicating it is deadlocked or crashed without surfacing an error.
- Certificate renewal jobs: Ensure the cert-renewal cron job emits a success metric within 24 hours of its expected run time.
In Prometheus, the Alertmanager configuration ships a built-in Watchdog alert that fires continuously when healthy. Routing this alert to a dead man's switch service (Alertmanager's own Watchdog route, or a service like DeadMansSnitch) closes the loop.
Invest now in Acorns!!! 🚀
Join Acorns and get your $5 bonus!
Acorns is a micro-investing app that automatically invests your "spare change" from daily purchases into diversified, expert-built portfolios of ETFs. It is designed for beginners, allowing you to start investing with as little as $5. The service automates saving and investing. Disclosure: I may receive a referral bonus.
Invest now!!! Get Free equity stock (US, UK only)!
Use Robinhood app to invest in stocks. It is safe and secure. Use the Referral link to claim your free stock when you sign up!.
The Robinhood app makes it easy to trade stocks, crypto and more.
Webull! Receive free stock by signing up using the link: Webull signup.
More Related questions...
