aws-cloudformation / aws-cloudformation/cloudformation-coverage-roadmap

[AWS::CloudWatch::Alarm] - [Enhancement] - PFR: Native suppression of initialization state transition actions

Open
#2,486 0 comments 3 reactions 0 assignees View on GitHub
enhancement
Dominant language
No language data
Stars
1.1k
Forks
62
PR merge metrics
No merged PRs in 30d

Description

### Name of the resource

AWS::CloudWatch::Alarm

### Resource name

_No response_

### Description

When a CloudWatch Alarm is created via any method (PutMetricAlarm-API, CloudFormation, CDK, Terraform, Pulumi, etc.), it always starts in `INSUFFICIENT_DATA` state. Once metric data becomes available, the alarm transitions to either `OK` or `ALARM`. These initial state transitions trigger all configured alarm actions (`AlarmActions`, `OKActions`, `InsufficientDataActions`), generating notifications unrelated to any actual operational issue.

This is a fundamental problem for **any IaC workflow** where alarms are deployed alongside the resources they monitor (ALB, ECS, RDS, etc.). The monitored resource is not yet ready when the alarm is created, so the alarm fires before the environment has reached a stable state.

Depending on the alarm configuration, this produces multiple unwanted notifications **per alarm**:

- **With `TreatMissingData: missing` (default):** The alarm stays in `INSUFFICIENT_DATA` until data arrives, then transitions to `OK` — triggering `OKActions` (a recovery notification for a problem that never existed). **1 false notification per alarm.**

- **With `TreatMissingData: breaching`:** The alarm transitions from `INSUFFICIENT_DATA` → `ALARM` (triggers `AlarmActions` for a non-existent problem), then once the resource becomes healthy: `ALARM` → `OK` (triggers `OKActions`). **2 false notifications per alarm.**

### How other monitoring platforms handle this

This is a well-understood problem in monitoring. Other major platforms have addressed it natively:

- **Grafana** provides a "Pending Period" (`for` parameter). An alert enters a `Pending` state instead of firing immediately. Only if the condition stays true for the full pending duration does the alert fire. This absorbs initial state transitions.

- **Icinga2** offers `times.begin` on notification objects, delaying the first notification for a configurable duration after a hard state is reached. Its soft/hard state concept requires consecutive failures before confirming a problem.

- **Azure Monitor** has no `INSUFFICIENT_DATA` state that triggers actions. Metric alerts only fire when the condition is actively met. Missing data for a new resource does not trigger an alert.

- **GCP Cloud Monitoring** works the same as Azure: alerting policies only fire when conditions are actively met. No initial noise.

### Why existing CloudWatch features do not solve this

- **`ActionsEnabled: false`** — Suppresses actions at creation, but there is no native way to automatically re-enable them after the initial transition. Requires a custom Lambda or CloudFormation Custom Resource to toggle it back.

- **`TreatMissingData`** — Controls how missing data points are evaluated but does not prevent the initial `INSUFFICIENT_DATA` state or subsequent transitions. Setting it to `breaching` makes the problem worse (2 false notifications instead of 1). Setting it to `ignore` does not help either: the alarm stays in `INSUFFICIENT_DATA` until data arrives and then fires the transition to `OK` or `ALARM` just the same.

- **Alarm Mute Rules (`AWS::CloudWatch::AlarmMuteRule`)** — When a mute window ends, CloudWatch re-triggers any alarm actions if the alarm had a state change while muted ([documented behaviour](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/alarm-mute-rules-behaviour.html)). **The initial transition notification is deferred, not suppressed.**

- **Composite Alarms with Suppressor Alarms** — The composite alarm itself starts in `INSUFFICIENT_DATA` and goes through the same initialization lifecycle. Does not solve the root cause. Additionally, composite alarms cost $0.50/month each compared to $0.10 for standard metric alarms — a workaround that increases per-alarm cost by 5x is not a viable solution.

- **Removing `OKActions`** — Not viable when downstream systems (monitoring dashboards, incident management tools) depend on `OK` notifications for recovery tracking.

- **SNS Subscription Filter Policies** — SNS supports filtering on message body fields (`FilterPolicyScope: MessageBody`), which could theoretically filter transitions from `INSUFFICIENT_DATA`. However, on a shared SNS topic serving all alarms in an account (including production), such a filter would suppress initialization transitions for **all** alarms — not just newly created ones. Legitimate `INSUFFICIENT_DATA` transitions on existing alarms (e.g., a metric stops publishing) would also be silently dropped.

### The only remaining workarounds require custom Lambda functions

The only options left are custom Lambda functions: either an EventBridge rule that checks the alarm creation timestamp before forwarding to SNS, or a CloudFormation Custom Resource that toggles `ActionsEnabled` after a delay.

Each workaround brings operational overhead: additional IAM roles, Lambda monitoring, error handling, retry logic, cold start latency, and deployment complexity — in every stack that contains alarms.

**This is disproportionate to the simplicity of the requirement: do not notify on state transitions that are artifacts of alarm initialization.**

### Use case

We deploy ephemeral environments (feature branch environments for acceptance testing) multiple times per day via CDK. Each environment includes CloudWatch Alarms monitoring ALB HealthyHostCount, ECS service health, and other metrics. Our alarms use `TreatMissingData: breaching` and configure both `AlarmActions` and `OKActions` on the same SNS topic, which feeds into centralized monitoring dashboards and incident management.

Every deployment produces a burst of false notifications — two per alarm with `TreatMissingData: breaching`. These are indistinguishable from real alerts. They trigger the same escalation paths, create tickets in incident management systems (many of which bill per incident), and send messages to Ops or Dev (MS-Teams-)Channels. The result is alert fatigue: teams learn to ignore notifications during deployments, which erodes trust in the monitoring system and increases the risk of missing real incidents.

This is not limited to ephemeral environments. It affects any scenario where infrastructure stacks containing alarms are deployed:

- Regular IaC pipelines — every `cdk deploy` or `aws cloudformation create-stack` that includes alarms
- Blue/Green Deployments — a new stack with its own alarm set for the green environment
- Disaster Recovery Failover — stacks with alarms deployed in a secondary region

At scale, false notifications become a cost factor (SNS pricing, downstream system load) and a reliability risk. Alert fatigue leading to missed real incidents is a well-documented anti-pattern ([Google SRE Book, Chapter 6: Monitoring Distributed Systems](https://sre.google/sre-book/monitoring-distributed-systems/)).

### Proposed solution

Add a property to `AWS::CloudWatch::Alarm` (and the `PutMetricAlarm` API) that suppresses alarm actions until the alarm has reached a stable, data-driven state.

**Option A — `SuppressActionsUntilDataDriven: true`:** (We think it's the best option)
All alarm actions are suppressed until the alarm has completed at least one full evaluation cycle with actual metric data points (not `INSUFFICIENT_DATA`). "Data-driven" means the evaluation used actual metric data points, not the `TreatMissingData` treatment of absent data — so an alarm with `TreatMissingData: breaching` that transitions to `ALARM` purely because no data points exist yet would still count as an initialization transition. Once the alarm reaches a data-driven `OK` or `ALARM` state, all subsequent state changes trigger actions normally. No time-based guessing required — works regardless of how long the resource takes to emit metrics. CloudWatch already tracks whether data points are present or absent during evaluation, so no new internal state tracking is needed.

**Option B — `ActionGracePeriodSeconds: `:**
All alarm actions are suppressed for N seconds after alarm creation. Simple and deterministic. Trade-off: requires the user to estimate how long their resource takes to stabilize.

Both options are opt-in and fully backward-compatible (default behaviour unchanged).

### Expected CloudFormation and CDK integration

This feature requires a new parameter on the `PutMetricAlarm` API — CloudFormation and CDK cannot implement this independently. Once the API parameter exists, we would expect it to be exposed as:

- A new property on the `AWS::CloudWatch::Alarm` CloudFormation resource type
- A corresponding property on the CDK `aws-cdk-lib/aws-cloudwatch.Alarm` L2 construct

A single declarative property would replace what today requires a multi-resource workaround: a CloudFormation Custom Resource backed by a Lambda function, an IAM execution role, and either an EventBridge rule or a Step Functions wait state — all to toggle `ActionsEnabled` after the initial state transitions. This workaround exists in every stack that contains alarms and must be maintained, monitored, and debugged independently.

With a native property, the entire alarm lifecycle — creation, suppression of initialization noise, and steady-state monitoring — would live in a single `AWS::CloudWatch::Alarm` resource definition. No additional resources, no custom code, no operational overhead.

### Other Details

**CloudWatch documentation:**
- [PutMetricAlarm API](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_PutMetricAlarm.html) — no relevant parameter exists today
- [CloudWatch Alarm Concepts](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html) — state transition behaviour
- [Alarm Mute Rules Behaviour](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/alarm-mute-rules-behaviour.html) — defers actions, does not suppress them

**Other platforms for reference:**
- [Grafana Pending Period (`for` parameter)](https://grafana.com/docs/grafana/latest/alerting/fundamentals/alert-rule-evaluation/#pending-period) — also available in [AWS Managed Grafana](https://docs.aws.amazon.com/grafana/latest/userguide/v10-alerting-rules-evaluation.html)
- [Icinga2 Notification Delay (`times.begin`)](https://icinga.com/docs/icinga-2/latest/doc/03-monitoring-basics/#notification-delay)

Contributor guide

Open the contributing guide

Research direction

Start with the linked PutMetricAlarm API and CloudWatch Alarm documentation, then review the expected AWS::CloudWatch::Alarm and CDK integration described in the issue. Compare the proposed data-driven and time-based suppression options, and consider how they preserve existing action behavior. Done would require an accepted API and CloudFormation resource design rather than a change to this roadmap repository.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, terraform
Domain
cloud, infrastructure, observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.