[Feature Request] "No Data" behavior setting for Metric Monitors
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 44.8k
- Forks
- 4.9k
- Avg merge
- 21h 23m
- Merged PRs (30d)
- 607
Description
Problem
When a Metric Monitor (metric alert rule) has no data points in an evaluation interval — e.g. a zero-traffic window overnight — the aggregation returns null. Sentry coerces that null to 0 before threshold evaluation, so a "below X" monitor fires a false alert on an empty window.
Critically, this coercion happens after the equation has already been evaluated in Snuba, which means formula-level workarounds cannot fix it (details below).
Root cause
src/sentry/incidents/utils/process_update_helpers.py:67-76
def get_aggregation_value_helper(subscription_update: QuerySubscriptionUpdate) -> float:
aggregation_value = list(subscription_update["values"]["data"][0].values())[0]
# In some cases Snuba can return a None value for an aggregation. This means
# there were no rows present when we made the query for certain types of aggregations
# like avg. Defaulting this to 0 for now. It might turn out that we'd prefer to skip
# the update in the future.
if aggregation_value is None:
aggregation_value = 0
return aggregation_value
The existing comment already anticipates this exact request: "It might turn out that we'd prefer to skip the update in the future." This issue is a request to make that configurable rather than hardcoded.
Note the return type is float, not float | None — so there is currently no way for a metric monitor to signal "no data, skip this update".
Precedent: this already exists for crash-rate alerts
In the same module, get_crash_rate_alert_metrics_aggregation_value_helper (lines 52-54) handles the empty case correctly:
if total_session_count == 0:
metrics.incr("incidents.alert_rules.ignore_update_no_session_data")
return None
So "skip the update when there's no data" is already implemented and shipping for one alert type. The request is to generalize it and make the policy configurable for metric monitors.
Impact
This pages humans on false positives, overnight, when traffic is legitimately zero.
Concrete example from our production setup: an alert fired at 02:38 CEST on 2026-07-05 on an overnight window with almost no traffic. There was no incident — traffic was simply near-zero, which is exactly when this monitor is least meaningful and most likely to fire.
Use case
We monitor SMS delivery rate for an OTP service, with a smoothing formula specifically designed to handle zero-traffic periods:
( sum_if(outcome:sent OR outcome:delivered, value, sms_delivery_report) + 10 )
/ ( sum(value, sms_delivery_report) + 10 )
The +10 smoothing is intended so that zero traffic evaluates to 10/10 = 1.0 (healthy).
It does not work. At the Snuba level the aggregates come back null (-OrNull), and null arithmetic propagates — null + 10 is null, null / null is null — so the equation returns null rather than 1.0. Sentry then coerces that null to 0, and 0 < 0.6 fires.
Requested feature
Add a "No Data Behavior" setting to Metric Monitors, as offered by comparable products — Datadog's "Notify if data is missing" (notify_no_data) and Grafana's "No data and error handling" (No Data → Alerting / No Data / OK).
Options:
- Treat as OK / Resolved ← most useful for our case
- Treat as Critical — for monitors where absence of data is the failure
- Keep last known state
- Skip the update — equivalent to today's crash-rate behavior
Default should remain the current behavior (coerce to 0) so existing monitors are unaffected and this is purely opt-in.
Workarounds considered
- Smoothing formula — does not work.
nullpropagates through the equation in Snuba before Sentry ever sees it, so the arithmetic never runs. This is the workaround most likely to be suggested, and it is structurally unavailable. - Emit a heartbeat metric with value
0— requires maintaining extra infrastructure and pollutes metric data with synthetic points. - Switch to Anomaly Detection — loses the explicit threshold we need for SLA tracking.
- Switch to a count-based threshold (e.g.
sum(all) - sum_if(sent OR delivered)above N/hr) — this does work and is our current fallback, but it loses the rate semantics. "Delivery rate below 60%" is the SLA we actually care about; a raw failure count is a proxy that needs re-tuning whenever traffic volume changes.
Expected behavior
When no data is present in an evaluation interval, the monitor should apply a configurable "no data" policy instead of defaulting to a threshold violation.
Environment
- Sentry SaaS
- Application Metrics (trace-connected), post-GA — per the update in #102275 stating alerting is now fully supported
Steps to reproduce
- Create a metric monitor with a percentage/rate equation over a custom metric, threshold "below X".
- Stop emitting the metric entirely (or wait for a natural zero-traffic window).
- Observe the monitor fire, with the aggregation evaluated as
0rather than being skipped or treated as healthy.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with src/sentry/incidents/utils/process_update_helpers.py, especially get_aggregation_value_helper and get_crash_rate_alert_metrics_aggregation_value_helper, then trace the metric monitor update path and its existing tests. Define how each requested no-data policy and the unchanged default should be represented and verified when an aggregation is None.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100