Cloud / Amazon CloudWatch Interview questions

Amazon CloudWatch is AWS's monitoring and observability service. It collects metrics, logs, and events from AWS resources, on-premises servers, and applications so you can track performance and health in one place.

Beyond collection, CloudWatch lets you visualize data on dashboards, set alarms that watch for threshold breaches, and trigger automated actions like scaling an Auto Scaling group or notifying a team through SNS.

It underpins most operational workflows in AWS because nearly every service - EC2, Lambda, RDS, ECS, and dozens more - publishes metrics and logs to it automatically, giving a consistent monitoring surface across an account.

A CloudWatch metric is a time-ordered set of data points representing the value of a variable over time, such as CPUUtilization or RequestCount.

Every metric belongs to a namespace, and most also carry one or more dimensions that describe the metric further, such as which instance or load balancer the value came from.

Metrics can come from AWS services automatically (like EC2 or RDS metrics) or be published by your own application code using the PutMetricData API, which is what makes a metric "custom".

A namespace is a container that isolates a set of metrics from one another so that metrics from different services or applications don't collide.

AWS services publish into predefined namespaces prefixed with AWS/, for example AWS/EC2 or AWS/Lambda.

When you publish custom metrics, you choose your own namespace name, but AWS reserves the AWS/ prefix, so a custom namespace can never start with those characters.

Dimensions are name/value pairs that further identify a metric within its namespace, such as InstanceId=i-0abc123 or FunctionName=my-func.

They let a single conceptual metric, like CPUUtilization, exist separately for every resource that reports it, and they let you filter or aggregate that metric by a specific dimension value.

A single metric can carry up to 30 dimensions, and CloudWatch treats any change in dimension name or value combination as a distinct metric stream for storage and pricing purposes.

A CloudWatch alarm watches a single metric, or the result of a metric math expression, over a period of time and compares it against a threshold you define.

Based on that comparison it holds one of three states, and on a state change it can invoke actions such as publishing to an SNS topic, executing an Auto Scaling policy, or stopping/rebooting an EC2 instance.

Alarms are the mechanism that turns passive metric collection into active, automated response, which is why they sit at the center of most CloudWatch-based operational workflows.

A CloudWatch alarm can be in one of three states at any given time.

Alarm actions are configured per state transition, so you can, for example, notify on entering ALARM and notify again separately when it returns to OK.

A dashboard is a customizable, at-a-glance view built from widgets - line graphs, number displays, text panels, alarm status, and even log query results.

You can pull metrics from multiple namespaces, regions, and (with cross-account observability enabled) multiple AWS accounts onto a single dashboard.

Dashboards are typically used as the shared operational view a team keeps open during incidents or reviews, since they refresh automatically and don't require querying each metric individually.

CloudWatch Logs is the log management component of CloudWatch: it centralizes log data from EC2 instances, Lambda functions, ECS tasks, VPC Flow Logs, Route 53 queries, and many other sources.

Once ingested, logs can be searched interactively, filtered into metrics, streamed to other services, and retained according to a policy you set per log group.

Because it's a managed service, you don't provision storage or indexing infrastructure yourself - you send log events and CloudWatch Logs handles durability and searchability.

A log group is a container that holds a related set of log streams, typically one log group per application or per AWS resource type, such as /aws/lambda/my-function.

Settings like retention period, KMS encryption, and access permissions are configured at the log group level and apply to every stream inside it.

Log groups are also the unit you query against in CloudWatch Logs Insights and the unit you attach subscription filters and metric filters to.

A log stream is a sequence of log events that all come from the same source, for example, a single EC2 instance, a single Lambda execution environment, or one ECS task.

It lives inside a log group and inherits that group's retention and permission settings, but each stream is ordered independently by the time its events were ingested.

In practice, if you run the same Lambda function across many concurrent invocations, you'll often see multiple log streams under one log group, one per underlying execution environment.

The CloudWatch agent is software you install on EC2 instances or on-premises servers to collect metrics and logs that AWS doesn't gather automatically, such as memory usage, disk space, and custom log files.

It's called "unified" because it replaced two older, separate agents - the original CloudWatch Logs agent and the SSM-based metrics collection - with a single agent and a single JSON configuration file.

You typically deploy and manage it through Systems Manager (SSM) Run Command, State Manager, or a configuration baked into an AMI, and it supports both Linux and Windows.

A custom metric is any metric your own application publishes to CloudWatch rather than one an AWS service generates automatically.

You publish it using the PutMetricData API (directly, via the CLI/SDK, through the CloudWatch agent's statsd/collectd support, or via Embedded Metric Format in logs), choosing your own namespace, metric name, and dimensions.

Common examples include business metrics like orders processed per minute or queue depth for an internal worker, things AWS has no way of knowing about on its own.

CloudWatch Logs Insights is an interactive query service for searching and analyzing log data stored in CloudWatch Logs, without needing to set up any separate indexing or ETL pipeline.

Queries use a purpose-built syntax with commands like fields, filter, stats, and sort, chained together with the pipe character.

fields @timestamp, @message
| filter @message like /ERROR/
| stats count() by bin(5m)

Results typically return in seconds even against large volumes, because CloudWatch scans the underlying compressed log data in parallel across the time range you specify.

CloudWatch Events was the original name for AWS's event-routing service: rules that match incoming events (or a schedule) and route them to targets like Lambda, SNS, or Step Functions.

That capability was expanded and rebranded as Amazon EventBridge, which added support for custom event buses, schema registry, and third-party SaaS event sources, while remaining backward compatible with existing CloudWatch Events rules.

In practice, "CloudWatch Events" rules still run on the default event bus in EventBridge, so the two names refer to the same underlying event-routing mechanism at different points in its evolution.

CloudWatch supports two metric resolutions: standard and high-resolution.

Standard resolution stores data points with one-minute granularity and is the default for most AWS service metrics.

High-resolution metrics store data points down to one-second granularity, are typically used for custom metrics that need faster reaction times, and allow alarms to evaluate on periods as short as 10 seconds, at a higher cost than standard resolution.

A composite alarm doesn't watch a metric directly - instead, it evaluates the states of other CloudWatch alarms using a rule expression built from AND, OR, and NOT operators.

For example, a composite alarm might only go into ALARM when both a latency alarm and an error-rate alarm are simultaneously in ALARM, filtering out cases where just one twitched briefly.

This is used to suppress noisy or correlated alerts, since it lets you express "alert me only when this combination of conditions holds" instead of getting one notification per underlying alarm.

CloudWatch Synthetics lets you create "canaries" - configurable scripts, written in Node.js or Python using Puppeteer/Selenium-style APIs, that run on a schedule to simulate real user or API traffic against your endpoints.

A canary can do something as simple as an HTTP GET and status check, or as involved as logging into a web app, clicking through a checkout flow, and verifying the final page content.

Each run publishes success/failure and duration metrics to CloudWatch and can capture screenshots and HAR files, giving you outside-in visibility into availability even when no real user has hit the endpoint recently.

By default, without installing any agent, EC2 automatically reports host- and hypervisor-level metrics under the AWS/EC2 namespace.

• CPUUtilization - percentage of allocated compute used
• NetworkIn / NetworkOut - bytes received/sent
• DiskReadOps / DiskWriteOps and DiskReadBytes / DiskWriteBytes
• StatusCheckFailed, StatusCheckFailed_Instance, StatusCheckFailed_System

Notably absent from this default list are memory utilization and disk space usage, since those require visibility inside the guest OS - which is exactly what the CloudWatch agent is for.

ServiceLens integrates AWS X-Ray traces, CloudWatch metrics, logs, and synthetic canary results into a single service map and drill-down view.

Instead of separately checking a metric dashboard, then a log group, then a trace, you can start from a visual map of your services, click a node showing elevated latency or errors, and pivot directly into the correlated traces and logs for that exact time window.

It's aimed at reducing the "tool-hopping" that normally slows down root-cause analysis in a distributed application.

A metric filter is a pattern you attach to a log group that CloudWatch evaluates against every incoming log event as it's ingested.

When an event matches the pattern - for example, a line containing "ERROR" or a JSON field like { $.statusCode = 500 } - CloudWatch increments a custom metric you've defined, optionally extracting a numeric value from the event instead of just counting matches.

You create one from the log group's console page or via put-metric-filter in the CLI, and once it's attached, the resulting metric behaves like any other CloudWatch metric and can back an alarm or a dashboard widget.

CloudWatch matters because it's the one monitoring surface that virtually every AWS service already writes to, so you get baseline visibility - metrics, logs, and health signals - without deploying extra collectors for most workloads.

It also closes the loop between detection and response: an alarm doesn't just tell a human something is wrong, it can directly trigger Auto Scaling, run an SSM automation document, or fan out through EventBridge, which shortens mean time to recovery.

That native, account-wide coverage is also why it's usually the starting point even in shops that later layer a third-party observability platform on top - CloudWatch either feeds that platform data (via Metric Streams or log subscriptions) or serves as the fallback when the third-party tool is down.

Manual monitoring - someone watching a dashboard - doesn't scale past a handful of resources, runs on human attention span, and has detection latency measured in however long until someone happens to look.

Alarms evaluate continuously against every configured metric, at whatever granularity you've chosen (down to 10 seconds for high-resolution metrics), so detection latency becomes a property of your configuration rather than a person's schedule.

More importantly, alarms can act, not just notify: tying an alarm to an Auto Scaling policy or an SSM automation means remediation can happen before anyone is even paged, which manual monitoring structurally cannot do.

A Logs Insights query is a sequence of commands separated by the pipe character, evaluated top to bottom against the log events in the log group(s) and time range you selected.

fields selects which fields to display, filter narrows events using boolean/regex conditions, stats aggregates (count, sum, avg, percentiles) optionally grouped by a field or time bucket via bin(), and sort/limit control ordering and row count.

fields @timestamp, @message
| filter @message like /5\d\d/
| stats count() as errorCount by bin(1h)
| sort errorCount desc

Because JSON log fields are auto-discovered, you can reference nested keys like $.requestId directly without predefining a schema, which is what makes ad hoc troubleshooting fast.

A standard alarm evaluates one metric, or one metric math expression, against a numeric threshold - it has no awareness of any other alarm.

A composite alarm evaluates a boolean rule expression built from the ALARM/OK states of other alarms, for example ALARM(HighLatency) AND ALARM(HighErrorRate), meaning its own state depends entirely on other alarms rather than on a metric directly.

The practical effect is suppression and correlation: a standard alarm fires independently every time its own condition is met, while a composite alarm can be designed to stay quiet unless several related conditions hold together, which is the main tool for cutting down alert noise in interdependent systems.

Use high-resolution metrics when the thing you're measuring changes meaningfully within a minute and a one-minute-old data point would be too stale to act on - classic examples are request rate for a bursty API or queue depth for a worker that needs to scale up in seconds, not minutes.

High resolution also unlocks alarm evaluation periods as short as 10 seconds, so an Auto Scaling policy tied to it can react to a spike well before the standard one-minute cadence would even register it.

The trade-off is cost and volume: high-resolution custom metrics are billed at a higher rate per metric than standard-resolution ones, so it's worth reserving for the specific metrics where sub-minute reaction genuinely changes an outcome, not applying it blanket across every custom metric you publish.

Choose CloudWatch Logs when you need the actual content of what happened - a stack trace, a specific error message, the payload that caused a validation failure - since logs carry arbitrary text and structured detail that a trace simply doesn't record.

Choose X-Ray when the question is about where time or errors are occurring across a chain of service calls, since it's built to show the timing breakdown and dependency graph of a single request as it hops between services.

In practice they're complementary rather than competing: a common pattern is finding a slow or failed request in X-Ray, grabbing its trace ID, then pivoting into CloudWatch Logs (or ServiceLens, which does this correlation for you) to read the exact log lines from that specific request.

CloudWatch and CloudTrail answer fundamentally different questions and are often confused because both are "logging" services in a loose sense.

In fact CloudTrail can deliver its event history into a CloudWatch Logs log group, letting you alarm on specific API activity, such as someone disabling a security group rule, using the same alarm mechanism you'd use for a performance metric.

Logs and metrics store fundamentally different shapes of data even though they share the same product name.

Because metric filters only move data one direction, a common design mistake is trying to reconstruct detailed event context from a metric graph after the fact - if you need that context later, it has to already exist in the underlying logs.

These two agents solve different problems, so "better" depends on what you're asking them to do - but for the specific job of collecting metrics and logs into CloudWatch, the CloudWatch agent is the right tool.

The CloudWatch agent is purpose-built to gather OS-level metrics (memory, disk, custom processes) and log files and ship them to CloudWatch; the SSM agent's job is remote command execution, patching, and automation - it can run commands that install or configure the CloudWatch agent, but it doesn't collect metrics itself.

The typical, and recommended, setup uses both together: SSM (via Run Command or State Manager) to deploy and keep the CloudWatch agent's configuration consistent across a fleet, while the CloudWatch agent itself does the actual metric and log collection.

Most CloudWatch bills are driven by three things: log ingestion/storage volume, the number of custom metrics and their resolution, and API call volume for polling data.

1. Set sensible log retention per log group instead of "never expire," and move rarely-accessed logs to the Infrequent Access log class.
2. Reduce log verbosity at the source (log level) rather than ingesting everything and filtering later.
3. Use metric filters to derive counts you need instead of ingesting entire verbose logs just to get a number.
4. Consolidate custom metrics using dimensions instead of creating many near-duplicate metric names.
5. Prefer Metric Streams over repeated GetMetricData/GetMetricWidgetImage polling if you're exporting to another system.
6. Delete unused log groups, dashboards, and alarms left over from decommissioned resources.

Reviewing the CloudWatch line item in Cost Explorer by usage type periodically will usually reveal which of these is the actual driver, rather than optimizing blind.

Start with permissions: confirm the role or user publishing the metric actually has cloudwatch:PutMetricData, since a silent permission failure is the single most common cause of "nothing shows up."

Next, verify the exact namespace, metric name, dimension names, and dimension values match what you're searching for in the console - CloudWatch treats these as case-sensitive and exact, so a mismatched casing or an extra dimension creates what looks like a missing metric but is really a different metric.

Check timing: some AWS service metrics only publish every five minutes under basic monitoring, and there can be a short propagation delay even for metrics that are present.

If you're using the CloudWatch agent, check its own local log file for errors - a misconfigured agent config JSON will fail to publish and log the reason locally, rather than raising anything visible from the console side.

An alarm's lifecycle starts the moment it's created against a metric (or metric math expression), a statistic, a period, and a threshold with a comparison operator.

From then on, CloudWatch continuously collects the underlying metric's data points and evaluates them at the end of each period against the "datapoints to alarm" setting - for example, requiring 3 out of the last 5 periods to breach before changing state, which avoids flapping on a single noisy point.

flowchart TD
  A["Alarm created: metric, threshold, period"] --> B["Metric data points collected"]
  B --> C{Evaluate over period vs datapoints-to-alarm}
  C -->|Breaches threshold| D["State: ALARM"]
  C -->|Within threshold| E["State: OK"]
  C -->|Not enough data| F["State: INSUFFICIENT_DATA"]
  D --> G["Configured actions fire: SNS, Auto Scaling, EC2 action"]
  E --> H["Actions fire only if configured for OK transition"]
  D --> I["Transition logged to alarm history"]
  E --> I
  F --> I

Once a state transition happens, CloudWatch invokes whatever actions are attached to that specific transition, and every transition is recorded permanently in the alarm's history, which is what you'd review after the fact to reconstruct exactly when and why an alert fired.

A subscription filter is attached to a log group and, unlike a metric filter, forwards the actual matching log data to a destination in near real time instead of just incrementing a metric.

sequenceDiagram
  participant App as Application
  participant LG as CloudWatch Log Group
  participant SF as Subscription Filter
  participant L as Lambda Function
  App->>LG: Write log event
  LG->>SF: Evaluate filter pattern
  SF-->>LG: Match found
  LG->>L: Invoke async with gzip+base64 batch
  L->>L: Decode and process records

Matching events are batched, compressed with gzip, and base64-encoded before Lambda receives them, so the very first thing the function must do is decode and decompress the payload before it can read individual log records.

The invocation is asynchronous, so if the function errors, CloudWatch Logs retries the delivery according to Lambda's standard async retry behavior rather than blocking the log group from accepting new events.

When you call PutMetricData, CloudWatch doesn't necessarily store every individual value you send as a separate row - within the same one-minute (or high-resolution one-second) window, it rolls values for the same metric/dimension combination into a statistic set: sample count, sum, minimum, and maximum.

When you later query or graph that metric over a longer period, say a one-hour period with the Average statistic, CloudWatch computes it from those underlying statistic sets rather than from raw individual values, which is why sum- and count-based statistics remain mathematically exact across aggregation while percentile-style statistics need extra internal handling to stay accurate.

This is also why publishing pre-aggregated statistic sets yourself (rather than one PutMetricData call per raw value) is both supported and often more efficient for high-throughput custom metrics, since it mirrors exactly what CloudWatch would have done internally anyway.

Most metrics are published and stored at standard, one-minute resolution by default, which already introduces up to a minute of inherent latency between something happening and a data point representing it.

Some AWS services only publish under "basic monitoring" every five minutes unless you explicitly enable detailed monitoring, which further widens the gap for those specific metrics.

On top of the publishing interval, there's a short propagation delay as the value moves from the source resource through the service's internal pipeline into the CloudWatch backend, and dashboards/alarms only ever evaluate against data points that have already made that full trip.

INSUFFICIENT_DATA means CloudWatch doesn't yet have enough data points within the evaluation window to confidently say OK or ALARM - this commonly happens right after an alarm is created, or when the underlying resource (and therefore its metric) stops reporting entirely.

By default, actions attached to the ALARM or OK transitions do not fire just because the state becomes INSUFFICIENT_DATA, unless you've explicitly attached an action to that specific transition.

The "treat missing data" setting on the alarm controls this precisely: you can choose missing (the default INSUFFICIENT_DATA behavior), notBreaching (treat gaps as if they were OK), breaching (treat gaps as if they were a threshold breach), or ignore (keep the alarm's current state unchanged during the gap).

Auto Scaling policies - both target tracking and step scaling - are driven directly by CloudWatch alarms watching a metric like CPUUtilization, request count per target, or a custom metric you publish yourself.

With target tracking, you specify a target value (say, 50% CPU) and CloudWatch/Auto Scaling manage the underlying alarms and scaling adjustments automatically; with step scaling, you define the alarms and the scaling adjustment per breach size yourself for finer control.

The relationship also runs in the other direction: the Auto Scaling group itself publishes metrics like GroupInServiceInstances and GroupDesiredCapacity back to CloudWatch, and scaling activities can be routed through EventBridge for further automation or notification.

Metric math lets you combine and transform multiple existing metrics into a new time series using an expression, without publishing a brand-new custom metric yourself.

A common example is computing an error rate from two raw metrics: m1 as 5xx count and m2 as total requests, then defining e1 = (m1/m2)*100 as the expression graphed or alarmed on.

It also powers built-in functions like ANOMALY_DETECTION_BAND(), and functions such as RATE(), SUM(), and AVG() across a set of metrics, all evaluated at query/alarm time rather than stored as a separate metric.

These represent two different philosophies for deciding when a metric is "wrong."

The practical guidance is to use static thresholds for metrics with a genuinely fixed acceptable range - like disk space or a hard SLA number - and reserve anomaly detection for metrics whose "normal" value legitimately shifts by time of day or day of week, where a single static number would either be too noisy or too insensitive.

Use Contributor Insights when you need to identify the top-N "contributors" inside high-cardinality log data - the noisiest client IPs, the URLs generating the most 5xx errors, the slowest API keys - without writing and maintaining a custom parsing script yourself.

You define a rule that tells CloudWatch which log fields to treat as the contributor dimension and which fields to aggregate, and it produces a time-series report and graph of the top contributors over any time range.

It's particularly useful during an active incident, where the immediate question is usually "who or what is causing this spike" rather than "what is the overall metric value," which a standard graph alone doesn't answer.

When you run a Logs Insights query, CloudWatch doesn't consult a pre-built index the way a traditional search engine would - it dispatches the scan across the compressed log data stored for the log group(s) and time range you specified, spreading the work across many parallel workers.

Each worker scans its assigned slice of log events, applies the filter stage first to discard non-matching events early, then performs field extraction (including auto-discovery of JSON keys) and any stats aggregation on the surviving subset, before results are merged and sorted centrally.

Because there's no persistent index to maintain, ingestion stays simple and cheap, but it also means query cost and latency scale with the volume of data scanned in the time range, not with the size of the result set - a narrow filter early in the query pipeline is what keeps a query against a huge log group fast, since it shrinks what later stages have to process.

Cross-log-group queries work by fanning the same scan-and-filter process out across each specified log group in parallel and merging results, rather than performing anything resembling a relational join between them.

Start in Cost Explorer filtered to the CloudWatch service, broken down by usage type, to confirm ingestion (DataProcessing-Bytes) rather than storage or API calls is actually the driver - these three cost components respond to very different fixes.

Once ingestion is confirmed, use the CloudWatch Logs "Log groups" usage view, or a Logs Insights query summing incoming bytes per log group, to find the specific noisy log groups rather than guessing across dozens of services.

Common root causes are a log level left at DEBUG in production, an application logging full request/response bodies, or a health-check endpoint logging on every single call - each of these is fixed at the source rather than after ingestion, since you're already paying for the bytes the moment they land in CloudWatch.

Where full log content genuinely needs to be kept but rarely queried, exporting older data to S3 (directly, or via the Logs Infrequent Access class) and shortening the CloudWatch retention window keeps the searchable, expensive tier smaller while preserving the raw history cheaply elsewhere.

CloudWatch cross-account observability (built on AWS Observability Access Manager) works by designating one account as the monitoring account and one or more others as source accounts.

sequenceDiagram
  participant M as Monitoring Account
  participant OAM as Observability Access Manager
  participant S as Source Account
  M->>OAM: Create sink (defines what can be shared)
  S->>OAM: Create link referencing the sink ARN
  OAM-->>S: Link established, sharing enabled
  M->>S: Query metrics/logs/traces read-only via unified console
  S-->>M: Return data (no role-switching required)

The monitoring account first creates a "sink" defining what telemetry types it's willing to receive; each source account then creates a "link" pointing at that sink's ARN, opting in to share its metrics, logs, and/or traces.

Once linked, the monitoring account can browse dashboards, run Logs Insights queries, and view traces from every linked source account inside a single console session, without assuming a role into each account individually, which is what makes it practical to operate observability centrally across a large AWS Organization.

EMF lets you emit metrics as part of a structured JSON log line instead of making a separate PutMetricData API call for every value.

The JSON payload includes your normal log fields plus a special _aws metadata block specifying the namespace, dimensions, and which fields in that same payload should be treated as metric values.

When CloudWatch Logs (directly, or via the Lambda extension/agent that supports EMF) ingests a log event containing that _aws block, it automatically extracts the specified fields and publishes them as CloudWatch metrics, in addition to storing the full JSON line as a normal, fully searchable log event.

The advantage is avoiding a second network call and its associated API cost/throttling risk for high-volume custom metrics - especially inside Lambda, where the extension buffers and flushes EMF-formatted metrics locally - while still retaining the complete contextual detail in the log itself for later Logs Insights queries.

CloudWatch Metric Streams continuously push nearly all metric updates to a Kinesis Data Firehose delivery stream as they're generated, typically landing within a few minutes and often faster, rather than you asking for them.

Polling with GetMetricData instead means your own system decides the interval, pays a per-request API cost, and risks being throttled if it's pulling a large number of metrics frequently - a pattern that gets expensive and fragile as the metric count grows into the thousands, which is exactly the scale many third-party observability platforms need to cover.

Because Metric Streams is a push model with output already formatted (JSON or the more compact OpenTelemetry-based binary format) for downstream consumption, it's the design AWS recommends specifically for continuously exporting most or all of an account's metrics to an external destination, reserving GetMetricData for targeted, occasional lookups rather than bulk continuous export.

A canary is packaged as a script plus a chosen runtime (Node.js with Puppeteer for browser-based checks, or a lighter runtime for simple HTTP checks) and stored so it can be re-executed on a defined schedule.

flowchart TD
  A["Schedule triggers run"] --> B["Managed Lambda environment starts"]
  B --> C["Canary script executes: HTTP call or headless browser flow"]
  C --> D["Assertions checked: status code, content, timing"]
  D --> E["Artifacts captured: screenshots, HAR file, logs"]
  E --> F["Artifacts stored in S3"]
  D --> G["Success/Failure + duration metrics published to CloudWatch"]
  G --> H["Canary run history updated"]
  G --> I{Alarm attached?}
  I -->|Threshold breached| J["Alarm fires"]

On each scheduled trigger, AWS spins up a managed Lambda execution environment behind the scenes, runs the script's steps, and evaluates whatever assertions you've coded, such as an expected status code or specific page text.

Regardless of pass or fail, the run publishes duration and success/failure metrics to CloudWatch and writes artifacts like screenshots and HAR files to an S3 bucket you configure, and if a CloudWatch alarm is attached to the failure metric, a string of failed runs will transition that alarm into ALARM just like any other metric-based alarm.

At small scale, one alarm per metric per service is manageable; at hundreds of services, the same pattern produces an alert flood where a single upstream failure trips dozens of unrelated-looking alarms simultaneously.

1. Use composite alarms to correlate dependent services, so a downstream alarm stays quiet when it's really just inheriting an already-alarming upstream failure.
2. Standardize metric dimensions and tagging conventions so alarms can be generated from a template in your IaC tool rather than hand-built per service, keeping configuration consistent as the fleet grows.
3. Prefer anomaly detection over static thresholds for services with genuinely variable, seasonal load, since a single static number tends to be either too noisy at low traffic or too insensitive at peak traffic.
4. Route all alarm state changes through EventBridge into a single incident-management pipeline instead of separate, ad hoc SNS topics per team, so on-call visibility stays centralized.
5. Build alarms around metric-math-derived SLO metrics (like error budget burn rate) rather than raw per-resource values, so the alert reflects business impact rather than infrastructure noise.

The common thread across all of these is shifting from "alert on every resource" toward "alert on correlated, business-relevant conditions," which is what actually keeps a large alarm surface usable during an incident.

Both measure front-end web performance, but from opposite directions - one observes real traffic, the other generates traffic on purpose.

Because RUM depends on real visitors, it can't tell you a page is broken before anyone hits it; Synthetics fills exactly that gap by proactively checking availability, which is why the two are typically deployed together rather than as alternatives to each other.

The four basic statistics - sample count, sum, minimum, and maximum - can be derived cleanly from an aggregated statistic set without keeping every individual raw value, which is why they aggregate perfectly across any time period.

A percentile like p99, however, requires knowledge of the actual distribution of values, not just those four summary numbers, so CloudWatch retains the finer-grained data needed to approximate percentiles when metrics are published, rather than only the collapsed statistic set.

Because of this, percentile statistics on custom metrics require enough underlying sample points to be statistically meaningful - CloudWatch enforces a minimum sample size for extended statistics on alarms specifically to avoid a percentile computed from just one or two data points being treated as reliable.

The practical implication is that percentile-based alarms are best applied to metrics with steady, reasonably high sample volume per period; a low-traffic custom metric evaluated at p99 over a short period can behave erratically simply because there isn't enough underlying data for the percentile to mean much.

The foundation is a dedicated monitoring account using CloudWatch cross-account observability (OAM), with every member account linked in as a source account so metrics, logs, and traces are queryable centrally without per-account role assumption.

For logs specifically, use subscription filters (or Firehose) to route selected log groups from member accounts into a central destination - either the monitoring account's own log groups, or an S3/OpenSearch pipeline - for workloads that need longer retention or cross-account search patterns beyond what OAM's live view covers.

Enforce consistency through Infrastructure as Code: standardized dimension and tagging conventions so alarms and dashboards can be templated identically across every account, rather than hand-built per team, and centrally-managed composite alarms and SLO-based metric math expressions that funnel into a single EventBridge-driven incident pipeline.

Where metrics need to leave AWS entirely - for a third-party platform, or long-term cross-account trend analysis - Metric Streams from each account (or centrally, once linked) avoids the API throttling and cost issues of polling every account individually, while Contributor Insights and Logs Insights remain the go-to tools for ad hoc, account-specific debugging that a standardized dashboard wasn't built to answer.