getsentry / getsentry/sentry

[Feature Request] "No Data" behavior setting for Metric Monitors

Open
#121,423 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Bug Product Area: Alerts Waiting for: Product Owner
Dominant language
Python
Stars
44.8k
Forks
4.9k
Avg merge
21h 23m
Merged PRs (30d)
607

Description

Problem

When a Metric Monitor (metric alert rule) has no data points in an evaluation interval — e.g. a zero-traffic window overnight — the aggregation returns null. Sentry coerces that null to 0 before threshold evaluation, so a "below X" monitor fires a false alert on an empty window.

Critically, this coercion happens after the equation has already been evaluated in Snuba, which means formula-level workarounds cannot fix it (details below).

Root cause

src/sentry/incidents/utils/process_update_helpers.py:67-76

def get_aggregation_value_helper(subscription_update: QuerySubscriptionUpdate) -> float:
    aggregation_value = list(subscription_update["values"]["data"][0].values())[0]
    # In some cases Snuba can return a None value for an aggregation. This means
    # there were no rows present when we made the query for certain types of aggregations
    # like avg. Defaulting this to 0 for now. It might turn out that we'd prefer to skip
    # the update in the future.
    if aggregation_value is None:
        aggregation_value = 0

    return aggregation_value

The existing comment already anticipates this exact request: "It might turn out that we'd prefer to skip the update in the future." This issue is a request to make that configurable rather than hardcoded.

Note the return type is float, not float | None — so there is currently no way for a metric monitor to signal "no data, skip this update".

Precedent: this already exists for crash-rate alerts

In the same module, get_crash_rate_alert_metrics_aggregation_value_helper (lines 52-54) handles the empty case correctly:

if total_session_count == 0:
    metrics.incr("incidents.alert_rules.ignore_update_no_session_data")
    return None

So "skip the update when there's no data" is already implemented and shipping for one alert type. The request is to generalize it and make the policy configurable for metric monitors.

Impact

This pages humans on false positives, overnight, when traffic is legitimately zero.

Concrete example from our production setup: an alert fired at 02:38 CEST on 2026-07-05 on an overnight window with almost no traffic. There was no incident — traffic was simply near-zero, which is exactly when this monitor is least meaningful and most likely to fire.

Use case

We monitor SMS delivery rate for an OTP service, with a smoothing formula specifically designed to handle zero-traffic periods:

( sum_if(outcome:sent OR outcome:delivered, value, sms_delivery_report) + 10 )
/ ( sum(value, sms_delivery_report) + 10 )

The +10 smoothing is intended so that zero traffic evaluates to 10/10 = 1.0 (healthy).

It does not work. At the Snuba level the aggregates come back null (-OrNull), and null arithmetic propagates — null + 10 is null, null / null is null — so the equation returns null rather than 1.0. Sentry then coerces that null to 0, and 0 < 0.6 fires.

Requested feature

Add a "No Data Behavior" setting to Metric Monitors, as offered by comparable products — Datadog's "Notify if data is missing" (notify_no_data) and Grafana's "No data and error handling" (No Data → Alerting / No Data / OK).

Options:

  • Treat as OK / Resolved ← most useful for our case
  • Treat as Critical — for monitors where absence of data is the failure
  • Keep last known state
  • Skip the update — equivalent to today's crash-rate behavior

Default should remain the current behavior (coerce to 0) so existing monitors are unaffected and this is purely opt-in.

Workarounds considered

  1. Smoothing formula — does not work. null propagates through the equation in Snuba before Sentry ever sees it, so the arithmetic never runs. This is the workaround most likely to be suggested, and it is structurally unavailable.
  2. Emit a heartbeat metric with value 0 — requires maintaining extra infrastructure and pollutes metric data with synthetic points.
  3. Switch to Anomaly Detection — loses the explicit threshold we need for SLA tracking.
  4. Switch to a count-based threshold (e.g. sum(all) - sum_if(sent OR delivered) above N/hr) — this does work and is our current fallback, but it loses the rate semantics. "Delivery rate below 60%" is the SLA we actually care about; a raw failure count is a proxy that needs re-tuning whenever traffic volume changes.

Expected behavior

When no data is present in an evaluation interval, the monitor should apply a configurable "no data" policy instead of defaulting to a threshold violation.

Environment

  • Sentry SaaS
  • Application Metrics (trace-connected), post-GA — per the update in #102275 stating alerting is now fully supported

Steps to reproduce

  1. Create a metric monitor with a percentage/rate equation over a custom metric, threshold "below X".
  2. Stop emitting the metric entirely (or wait for a natural zero-traffic window).
  3. Observe the monitor fire, with the aggregation evaluated as 0 rather than being skipped or treated as healthy.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/sentry/incidents/utils/process_update_helpers.py, especially get_aggregation_value_helper and get_crash_rate_alert_metrics_aggregation_value_helper, then trace the metric monitor update path and its existing tests. Define how each requested no-data policy and the unchanged default should be represented and verified when an aggregation is None.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.