Tools / Monitoring and Observability Interview Questions
1. What is the difference between monitoring and observability?
Monitoring and observability are related but distinct concepts. Monitoring is the practice of collecting predefined metrics, logs, and alerts to track whether a system is behaving as expected. You decide upfront what to watch — CPU usage, request rate, error count — and dashboards or alerts fire ...
2. What are the three pillars of observability?
The three pillars of observability are metrics , logs , and traces . Together they give operators three different lenses through which to understand system behavior. Metrics are numeric time-series data — counters, gauges, and histograms. They are cheap to store and query at scale, making them id...
3. What is a Service Level Indicator (SLI) and how does it differ from an SLO and SLA?
An SLI (Service Level Indicator) is a specific, measurable signal that reflects user experience — typically a ratio or rate. Common SLIs include availability (percentage of successful HTTP requests), latency (fraction of requests served under 200 ms), and error rate (5xx responses divided by tota...
4. What is an error budget and how is it used in SRE?
An error budget is the allowable amount of unreliability a service can have within a given SLO window. If your SLO promises 99.9% availability over 30 days, you have 0.1% of that window to spend on failures — roughly 43.2 minutes of downtime. That 43.2 minutes is your error budget. The budget is ...
5. What is distributed tracing and how does it work?
Distributed tracing is a technique for following a single request as it moves through multiple services in a distributed system. Without it, when a user reports slowness, you might see a problem in Service C but have no idea whether Service A or B caused it. The mechanism works through context pr...
6. What is OpenTelemetry and why has it become the industry standard?
OpenTelemetry (OTel) is a vendor-neutral, open-source observability framework that provides APIs, SDKs, and a collector for generating, collecting, and exporting telemetry data — metrics, logs, and traces — from applications. It was formed in 2019 by merging OpenCensus (Google) and OpenTracing (C...
7. What is the RED method for monitoring microservices?
The RED method, introduced by Tom Wilkie, defines three golden signals specifically suited to request-driven microservices: R — Rate : The number of requests per second the service is receiving. This tells you about load and traffic patterns. Sudden drops can indicate that upstream services stopp...
8. What are the Four Golden Signals defined by Google SRE?
Google's SRE book defines four signals that, when monitored together, give a comprehensive picture of a user-facing service's health: 1. Latency — The time it takes to serve a request. Critically, you must distinguish latency of successful requests from latency of failed requests. A 500 error tha...
9. What is Prometheus and how does its pull-based scraping model work?
Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud and now a CNCF graduated project. It stores all data as time-series: streams of timestamped float64 values identified by a metric name and a set of key-value labels. What makes Prometheus distincti...
10. What is Grafana and how does it integrate with Prometheus?
Grafana is an open-source analytics and visualization platform that lets you query, visualize, and alert on metrics from a wide variety of data sources — Prometheus, Loki, Tempo, InfluxDB, Elasticsearch, CloudWatch, and many more — all from a single UI. The integration with Prometheus works throu...
11. What is structured logging and why is it preferred over plain-text logs?
Structured logging is the practice of emitting log records as machine-parseable data — typically JSON — rather than free-form text strings. Each log entry is a document with well-defined fields: timestamp , level , message , service , trace_id , user_id , and any other contextual fields relevant ...
12. What is log aggregation and what tools are commonly used for it?
Log aggregation is the process of collecting log data from many sources — application instances, containers, VMs, serverless functions — into a centralized system where it can be searched, analyzed, and retained. Without aggregation, debugging a failure across 50 pods means SSHing into each one i...
13. What is alerting fatigue and how can you reduce it?
Alerting fatigue occurs when on-call engineers receive so many alerts — many of which are non-actionable, duplicate, or transient — that they begin ignoring or acknowledging them without investigation. It is one of the most damaging failure modes in an observability program because it means real ...
14. What is the USE method and when should you apply it?
The USE method was defined by Brendan Gregg as a systematic way to analyze performance problems in any system resource. USE stands for: U — Utilization : The percentage of time the resource is busy. A CPU at 90% utilization is heavily loaded. Disk at 100% utilization (100% of I/O time spent servi...
15. What is cardinality in metrics and why does high cardinality cause problems?
Cardinality in metrics refers to the number of unique label value combinations that a metric can produce. A metric like http_requests_total{method, status_code, endpoint} with 5 methods, 20 status codes, and 1,000 endpoints generates up to 100,000 unique time-series. Each unique combination is ca...
16. What is tail-based sampling in distributed tracing and when should you use it?
Tail-based sampling is a tracing strategy where the decision about whether to keep or discard a trace is made after the entire trace is complete, not at the moment the root span starts. This contrasts with head-based sampling , where a random coin flip at the entry point determines whether the tr...
17. What is a health check endpoint and what should it return?
A health check endpoint is an HTTP endpoint — typically /health , /healthz , or /actuator/health — that exposes the current health status of a service. Load balancers, orchestrators like Kubernetes, and monitoring systems poll this endpoint to determine whether the service is ready to receive tra...
18. What is synthetic monitoring and how does it differ from real user monitoring (RUM)?
Synthetic monitoring (also called active monitoring) involves simulating user interactions with your application using scripted probes that run on a schedule, independent of real user traffic. The probes check that key user journeys — login, checkout, search — work correctly and measure their per...
19. What are Core Web Vitals and why do they matter for observability?
Core Web Vitals are a set of user-experience metrics defined by Google that measure loading performance, interactivity, and visual stability. They are directly included in Google's search ranking algorithm, making them both an observability concern and a business one. The three current Core Web V...
20. What is application performance monitoring (APM) and how does it differ from infrastructure monitoring?
Application Performance Monitoring (APM) focuses on the behavior and performance of your application code — transaction tracing, method-level timing, database query performance, external API call latency, memory allocations, and error rates at the code level. APM tools like Datadog APM , New Reli...
21. What is eBPF and how is it revolutionizing observability?
eBPF (extended Berkeley Packet Filter) is a Linux kernel technology that allows sandboxed programs to run inside the kernel without modifying kernel source code or loading kernel modules. Originally designed for network packet filtering, eBPF has been extended to support arbitrary kernel and user...
22. What is Jaeger and how does it work as a distributed tracing backend?
Jaeger is an open-source distributed tracing platform originally developed by Uber and now a CNCF graduated project. It collects, stores, and visualizes distributed traces from microservices, making it possible to reconstruct the end-to-end journey of any request. Jaeger's architecture consists o...
23. What is MTTR and MTTD and why do they matter to SRE teams?
MTTR (Mean Time To Recover) and MTTD (Mean Time To Detect) are reliability engineering metrics that quantify two key phases of an incident lifecycle. MTTD — Mean Time To Detect is the average time between when a failure actually begins and when the monitoring system (or a customer) first detects ...
24. What is anomaly detection in observability and what are its limitations?
Anomaly detection in observability is the automated identification of data points or patterns in metrics, logs, or traces that deviate significantly from historical baselines or expected behavior. Instead of manually setting static thresholds like "alert if CPU > 80%", anomaly detection learns se...
25. What is a runbook and how should it be linked to monitoring alerts?
A runbook (also called a playbook) is a documented set of procedures that an on-call engineer follows when a specific alert fires. A well-written runbook dramatically reduces MTTR by pre-answering the first questions an engineer asks: What is this alert? Why does it matter? What checks do I run f...
26. What is a service mesh and how does it enhance observability?
A service mesh is an infrastructure layer — deployed alongside your application services — that manages service-to-service communication. It intercepts network traffic using sidecar proxies (Envoy is the most common) injected into every pod, handling load balancing, mutual TLS, retries, circuit b...
27. What is a postmortem and what makes one blameless?
A postmortem (also called an incident review or retrospective) is a structured document written after a significant incident. Its purpose is to understand what happened, why it happened, what impact it had, and how to prevent recurrence. In SRE culture, postmortems are treated as a learning oppor...
28. What is the difference between blackbox monitoring and whitebox monitoring?
Blackbox monitoring treats the system under observation as a black box — you probe it from the outside and measure what you can observe without any inside knowledge. You send HTTP requests to an endpoint and measure whether you get a 200 response and in what time. The Prometheus Blackbox Exporter...
29. What is Kubernetes monitoring and what are the key components to observe?
Kubernetes monitoring covers multiple layers, each requiring different tooling and instrumentation. A Kubernetes cluster has at minimum these observable layers: Control plane components : The API server, etcd, scheduler, and controller manager each expose their own Prometheus metrics. API server ...
30. What is a metric histogram and why is it used for latency measurement?
A histogram is a metric type that samples observations and counts them into configurable buckets, while also tracking a running count and sum. In Prometheus, a histogram metric creates multiple time-series: _bucket{le="0.1"} (count of observations ≤ 100 ms), _bucket{le="0.5"} , _bucket{le="1.0"} ...
31. What is chaos engineering and how does it relate to observability?
Chaos engineering is the practice of intentionally injecting failures into a system in a controlled way to discover weaknesses before they cause unplanned outages. The discipline was pioneered by Netflix with their Chaos Monkey tool (now part of the Simian Army), which randomly terminated EC2 ins...
32. What is log sampling and when should you apply it?
Log sampling is the practice of recording only a fraction of log entries that match a certain pattern, rather than every single one. It is a strategy for controlling log volume and cost when some log types are emitted at very high rates and provide diminishing marginal value per entry. The most c...
33. What is the difference between push-based and pull-based metrics collection?
In a pull-based system, the monitoring server (like Prometheus) periodically initiates HTTP requests to each target's metrics endpoint and fetches the current metric values. The monitoring server controls the scrape interval and decides which targets to scrape. In a push-based system, application...
34. What is distributed systems observability and what challenges does it introduce compared to monolith observability?
Distributed systems observability refers to the ability to understand the internal state and behavior of a system that consists of multiple independently deployed services communicating over a network. Unlike a monolith — where all code runs in one process and profiling, logging, and debugging ar...
35. What is Datadog and what differentiates it from open-source observability stacks?
Datadog is a cloud-based monitoring and observability platform that provides infrastructure monitoring, APM, log management, real user monitoring, synthetic testing, security monitoring, and more — all integrated in a single SaaS product. It is one of the dominant commercial observability platfor...
36. What is on-call rotation and what makes an on-call experience sustainable?
An on-call rotation is a scheduled arrangement where engineers take turns being the primary responder for production incidents outside normal business hours. When an alert fires, the on-call engineer receives a page (via PagerDuty, Opsgenie, or VictorOps) and is expected to acknowledge and begin ...
37. What is continuous profiling and how does it differ from traditional profiling?
Continuous profiling is the practice of running lightweight profilers in production continuously (24/7), sampling CPU usage, memory allocations, goroutine counts, or mutex contention at low frequency, and storing the results in a queryable database. The key word is continuously — unlike tradition...
38. What is a flame graph and how do you read it?
A flame graph is a visualization of a stack trace profile that makes it easy to identify which functions consume the most CPU time, memory, or other resources. It was invented by Brendan Gregg while at Netflix to visualize perf(1) output on Linux systems. Reading a flame graph: Y-axis (vertical) ...
39. What is the role of an observability platform in incident response?
An observability platform serves as the central nervous system of incident response. When an alert fires, the on-call engineer opens the platform and uses it through every phase of the incident lifecycle. Detection phase : Alerts integrated with PagerDuty or Opsgenie fire when SLO burn rates exce...
40. What is OpenMetrics and how does it relate to Prometheus exposition format?
OpenMetrics is a specification for transmitting metrics at scale that evolved from the Prometheus text exposition format. It was accepted as a CNCF sandbox project and aims to be the standard for metrics exposition across the industry, not just within the Prometheus ecosystem. The original Promet...
41. What is a dead man's switch alert and when should you use it?
A dead man's switch alert (also called a heartbeat alert or watchdog alert) is an alert that fires when it stops receiving a signal, rather than when it detects a problem. The pattern inverts the usual alerting logic: instead of "alert when metric X exceeds threshold Y", it says "alert if I have ...
42. What is Thanos and how does it extend Prometheus for large-scale deployments?
Thanos is an open-source, CNCF incubating project that extends Prometheus to provide highly available, long-term metrics storage at scale. It was created at Improbable (now Grafana Labs contributes heavily) and addresses two fundamental limitations of standalone Prometheus: single-node storage li...
43. How does observability apply to event-driven and asynchronous architectures?
Observability in event-driven architectures (EDA) — systems built around message queues like Apache Kafka, RabbitMQ, or AWS SQS — presents distinct challenges because requests do not follow a synchronous request-response path. A single business transaction might produce events consumed by multipl...
44. What is the difference between an alert and a notification in observability?
The terms "alert" and "notification" are often used interchangeably but represent different stages in the incident response pipeline. Understanding the distinction helps design more effective on-call systems. An alert is the detection event itself — the result of evaluating a rule against metric ...
45. What is observability-driven development (ODD) and how does it shift monitoring left?
Observability-driven development (ODD) is a practice where engineers write instrumentation — metrics, logs, and trace spans — as a first-class part of feature development, not as an afterthought added after a service is deployed. The principle is "if you cannot observe it, you cannot reason about...
