Tools / Datadog Interview questions
1. What is Real User Monitoring (RUM) in Datadog?
Real User Monitoring captures actual visitor sessions on your web or mobile app - page loads, clicks, navigation, resource timing, and errors - directly from the browser or device, rather than inferring user experience from backend metrics alone. A small...
2. What is Datadog Database Monitoring?
Database Monitoring (DBM) is a Datadog product that gives query-level visibility into databases like PostgreSQL, MySQL, and SQL Server - showing which specific queries are slow, their execution plans, and what they're waiting on. It's collected by a lightweight database...
3. What is Network Performance Monitoring in Datadog?
Network Performance Monitoring (NPM) shows network traffic between hosts, containers, and services - connections, throughput, retransmits, and latency - tagged the same way as the rest of your Datadog data, so you can see which service-to-service connection is actually degraded....
4. What is Datadog Serverless Monitoring?
Serverless Monitoring extends Datadog's metrics, traces, and logs to serverless functions - most commonly AWS Lambda, but also Azure Functions and Google Cloud Functions - where there's no host to install a traditional Agent on. It's typically enabled via a...
5. Describe the Datadog Cluster Agent?
The Cluster Agent is a specialized Kubernetes-aware component that sits between the Kubernetes API server and the per-node Datadog Agents, acting as a single, centralized point of contact for cluster-level data instead of every node querying the API server directly....
6. What is Datadog CI Visibility?
CI Visibility gives observability into continuous integration pipelines - build and test durations, failure rates, and flaky tests - across providers like Jenkins, GitLab, CircleCI, and GitHub Actions, in one consistent view. It ingests pipeline and job-level events plus individual...
7. What is Datadog Error Tracking?
Error Tracking automatically detects, groups, and deduplicates errors coming from logs, APM traces, and RUM sessions into a single trackable issue , instead of leaving every occurrence as a separate, unrelated log line. Each issue shows a stack trace, first-seen...
8. What is Continuous Profiler in Datadog?
Continuous Profiler is an always-on, low-overhead code profiler that runs in production, capturing where CPU time, memory allocations, and lock contention are actually being spent inside your application's code - down to the specific function or line. Unlike a one-off...
9. Describe Datadog Incident Management?
Incident Management is Datadog's built-in tooling for declaring, coordinating, and documenting an incident from detection through resolution, without needing a separate standalone tool. Declaring an incident creates a structured record with severity, an assigned commander, and a live timeline, an
10. What is Datadog Cloud Cost Management?
Cloud Cost Management ingests billing data from your cloud providers (like an AWS Cost and Usage Report) and correlates it with the same tags already used across your infrastructure monitoring, so cost can be broken down by team, service, or...
11. What are API keys and application keys in Datadog?
An API key identifies your Datadog organization and authorizes data to be submitted to it - it's what the Agent and integrations use to send metrics, traces, and logs in. An application key , by contrast, is tied to an...
12. What is the Datadog Terraform provider used for?
The Datadog Terraform provider lets you define dashboards, monitors, synthetic tests, notebooks, and other Datadog resources as version-controlled configuration files, applied through standard Terraform workflows instead of manual UI changes. This makes Datadog configuration reviewable through no
13. What is an outlier monitor in Datadog?
An outlier monitor flags a single host, container, or other entity that's behaving statistically different from its peer group - for example, one host in a fleet of fifty with unusually high latency while the rest look normal. Rather than...
14. What is a forecast monitor in Datadog?
A forecast monitor projects a metric's future values based on its historical pattern - using linear or seasonal forecasting algorithms - and alerts when the predicted value is likely to breach a configured threshold, not just when the current value...
15. What is the Datadog Service Catalog?
The Service Catalog is a centralized registry of every service running across your organization, showing ownership, on-call contacts, dependencies, documentation links, and a health or quality score for each one. It's populated either from YAML definition files checked into a...
16. Define OpenTelemetry support in Datadog?
Datadog can ingest telemetry (traces, metrics, and logs) produced using the vendor-neutral OpenTelemetry standard, either by sending data directly to the Agent's built-in OTLP receiver or by routing it through an OpenTelemetry Collector configured with the Datadog exporter. This means...
17. What is an Agent flare in Datadog?
An Agent flare is a diagnostic bundle - generated with a single command - that packages the Agent's logs, configuration (with secrets automatically redacted), and status information into a single archive. It's the standard way to hand troubleshooting context to...
18. What is Sensitive Data Scanner in Datadog?
Sensitive Data Scanner inspects telemetry - most commonly logs - for patterns matching sensitive data, such as credit card numbers, social security numbers, or custom-defined patterns, using either Datadog's built-in pattern library or custom regular expressions. When a match is...
19. Describe Datadog Workflow Automation?
Workflow Automation is a low-code tool for building automated sequences of actions - chaining API calls, notifications, and approvals - typically triggered by a monitor firing or an incident being declared. A workflow might, for example, automatically restart a service,...
20. What is Application Security Management in Datadog?
Application Security Management (ASM) detects and can block application-layer attacks - like SQL injection or cross-site scripting - by observing requests at runtime using the same tracing instrumentation already deployed for APM. Because it reuses existing APM instrumentation rather than...
21. What is the difference between API keys and application keys?
An API key authorizes data to flow into a Datadog organization - it's org-scoped, used by the Agent and integrations, and by itself doesn't grant any particular user's permissions for reading or managing resources. An application key is scoped to...
22. How does the Cluster Agent differ from the node-level Datadog Agent?
The node-level Agent runs as a DaemonSet, one pod per node, collecting host, pod, and container metrics, logs, and traces local to that specific node - it's the workhorse doing per-node data collection. The Cluster Agent runs as a small,...
23. Why do we use monitor mute/downtime instead of deleting a monitor?
Downtime silences a monitor's notifications for a defined time window - useful during planned maintenance - while leaving the monitor itself fully active: it keeps evaluating, keeps recording history, and simply suppresses the alert from actually notifying anyone during that...
24. What is the difference between Error Tracking and standard log-based error monitoring?
Standard log-based error monitoring treats every error occurrence as its own separate log line - to understand whether an error is new, worsening, or recurring, someone has to manually search and eyeball the pattern across many individual entries. Error Tracking...
25. How does Datadog's Continuous Profiler collect data without high overhead?
Rather than instrumenting every function call (which would add meaningful latency to every request), the profiler uses statistical sampling - periodically capturing a snapshot of what the application's threads are doing, at a configurable interval - and reconstructs a representative...
26. When should you use an outlier monitor versus a threshold monitor?
Use a threshold monitor when you know the exact acceptable value in advance and it's the same for every entity being monitored - like 'any host with disk usage over 90% is a problem,' regardless of how its peers are...
27. What is the difference between a process monitor and a network monitor in Datadog?
A process monitor tracks whether a specific process (matched by name or command-line pattern) is running on a host, and can alert on it stopping unexpectedly or on that process's own resource usage - it's fundamentally about a single process's...
28. How does Datadog ingest OpenTelemetry data?
There are two common ingestion paths: sending OTLP (OpenTelemetry Protocol) data directly to the Datadog Agent's built-in OTLP receiver, or routing it through a self-managed OpenTelemetry Collector configured with the Datadog exporter, which then forwards it to Datadog's intake. Either...
29. Why is Metrics without Limits useful for cost control?
By default, every tag combination submitted on a custom metric becomes a separate, billed timeseries - so a high-cardinality tag can inflate cost dramatically even if nobody ever actually queries by that tag. Metrics without Limits decouples ingestion from indexing...
30. What happens when Sensitive Data Scanner detects a match?
The scanner applies whichever action the matching rule is configured with: redact (replace the sensitive value with a placeholder), hash (replace it with a non-reversible hash, preserving the ability to spot repeated occurrences without exposing the value), or simply flag/tag...
31. How does Datadog's Cloud Cost Management attribute spend?
It ingests detailed billing export data from cloud providers, such as an AWS Cost and Usage Report, which breaks charges down to the individual resource level rather than a single lump sum per account. That resource-level billing data is then...
32. When should you use APM trace retention filters versus sampling rules?
Sampling rules operate earlier, at the client/Agent level, determining what percentage of traces are even generated and sent to Datadog in the first place - they control ingestion volume and cost, and also affect the trace-based metrics computed from what's...
33. What is the difference between Service Level Indicators and Service Level Objectives?
A Service Level Indicator (SLI) is the actual measured signal - for example, the ratio of successful requests to total requests over some period - it's a number reflecting what's really happening. A Service Level Objective (SLO) is the target...
34. How does Fleet Automation manage Agent upgrades across a fleet?
Fleet Automation gives a centralized view of every Agent's version, configuration, and health across an entire fleet, so operators don't have to manually check each host individually to know what's out of date. It leverages Datadog's remote configuration capability to...
35. Why is the Service Catalog important for large engineering organizations?
As the number of services grows into the hundreds, no single engineer can reliably remember who owns what, which services depend on which, or who to page - information that used to live in someone's head or an out-of-date wiki...
36. What is the difference between mobile RUM and browser RUM?
Browser RUM, via a JavaScript SDK, tracks page views, resource load timing, and browser-specific quality signals like Core Web Vitals - metrics that are meaningful in the context of a web page's lifecycle. Mobile RUM, via native SDKs for iOS,...
37. How does log rehydration work from Datadog archives?
Logs sent to a cloud storage archive (rather than fully indexed) aren't searchable in the Log Explorer by default - rehydration is the process of pulling a specific subset back out of that archive, based on a time range and...
38. When should you use dashboards-as-code instead of the UI editor?
Use dashboards-as-code (via the Terraform provider or the dashboards API with exported JSON) when a dashboard needs to be reviewed through a pull request, replicated identically across multiple environments or accounts, or kept in sync with an infrastructure-as-code workflow that...
39. Explain the execution flow of a RUM session being recorded and ingested?
When a page loads (or a mobile app launches), the RUM SDK initializes and starts a new session, generating a unique session ID that will tag every event produced during that session's lifetime. As the user interacts with the app...
40. How can you optimize APM costs using retention filters?
Start from the fact that ingestion and indexing are billed differently: sampling controls how much is generated/ingested at all, while retention filters control what stays searchable for the 15-day window - the default Intelligent Retention Filter already keeps errors and...
41. How do you troubleshoot a Database Monitoring integration reporting no query metrics?
First confirm the database-level prerequisite: DBM relies on database introspection views and settings (for example, ensuring query text capture is enabled, like track_activity_query_size on PostgreSQL) - if these aren't enabled at the database level, the Agent has nothing to read...
42. Explain the internal working of Cloud Workload Security (CWS)?
CWS uses eBPF probes attached to relevant kernel syscalls - process execution, file access, network activity - to observe what's actually happening on a host or inside a container at runtime, with low overhead since the observation happens in-kernel rather...
43. How can you optimize Kubernetes monitoring using the Cluster Agent's Cluster Checks?
Identify checks that are redundant when run per-node - typically checks against an external endpoint, a load-balanced Kubernetes Service, or a managed external database - since running the same check independently on every node Agent wastes resources checking the exact...
44. Explain the lifecycle of an incident in Datadog Incident Management?
An incident is declared - either manually by a responder or automatically triggered from a monitor - creating a structured record with an initial severity, a designated commander, and an auto-provisioned communication channel (commonly a dedicated Slack channel) to centralize...
45. Which is better for reducing MTTR: Watchdog RCA or manual root cause analysis, and why?
This isn't really an either-or choice in practice - they operate at different speeds and with different strengths, and the strongest approach uses Watchdog RCA to shortcut the slowest part of manual investigation rather than replacing manual judgment entirely. Watchdog...
46. How do you troubleshoot missing spans from an OpenTelemetry-instrumented service?
First confirm the receiving side is actually configured to accept OTLP: check that the Agent's OTLP receiver (or the Collector's Datadog exporter) is enabled on the expected port and protocol (gRPC vs HTTP), since a protocol or port mismatch between...
47. Explain the execution flow of Sensitive Data Scanner across logs and APM?
For logs, scanning happens as a stage within the log processing pipeline: after earlier processors (like the Grok Parser) have extracted structured fields, configured scanning rules are evaluated against the log's content and matching attributes, and any configured action -...
48. How can you optimize serverless monitoring for Lambda cold starts?
Prefer the Lambda extension/layer approach for instrumentation over a heavier wrapper that fully re-initializes tracing logic on every cold start, since the extension runs as a lightweight, separate process alongside the function and adds materially less to init duration than...
49. Explain the internal working of Datadog's remote configuration feature?
Each Agent maintains a persistent, authenticated connection (or regular polling cadence) to Datadog's remote configuration backend, checking for configuration updates that apply to it based on its own tags and current configuration state. When an operator pushes a change -...
50. How do you troubleshoot inconsistent cost attribution in Cloud Cost Management?
Start by checking the completeness of the underlying billing export itself - confirm the cloud provider's cost and usage report (or equivalent) is generating complete files without gaps, since a partial or delayed export will show as a genuine drop...
