Tools / Monitoring and Observability Interview Questions
What is on-call rotation and what makes an on-call experience sustainable?
An on-call rotation is a scheduled arrangement where engineers take turns being the primary responder for production incidents outside normal business hours. When an alert fires, the on-call engineer receives a page (via PagerDuty, Opsgenie, or VictorOps) and is expected to acknowledge and begin investigating within a defined response time (typically 5–15 minutes).
On-call is sustainable when several conditions are met:
Low alert volume: If the on-call engineer is paged more than a few times per shift, something is wrong with the alerting system. Google's SRE book recommends that on-call engineers spend no more than 25% of their time on operational work (toil). Frequent pages beyond that must trigger toil-reduction efforts.
Meaningful alerts: Every page should require a human decision. If an alert resolves itself without any action, it is either too sensitive or should auto-remediate. Pages that wake engineers at 3 AM for events that do not require action destroy morale and trust in the system.
Compensation: On-call work should be compensated — either financially (on-call pay) or with compensatory time off after a heavy on-call shift.
Escalation paths: The on-call engineer should not be alone. A clear secondary on-call, escalation contacts, and runbooks ensure that no single engineer is expected to know everything.
Post-incident investment: Each incident that required manual intervention is a toil-reduction opportunity. Sustainable on-call requires a cultural commitment to fix root causes rather than repeatedly firefighting the same issues.
More Related questions...