getsentry / getsentry/sentry

Use the original baseline for % based alerts

Open
#66,476 2 comments 0 reactions 0 assignees View on GitHub
Feature
Dominant language
Python
Stars
44.8k
Forks
4.9k
Avg merge
22h 21m
Merged PRs (30d)
586

Description

### Environment

SaaS ([https://sentry.io/]())

### Steps to Reproduce

Today, %-based metric alerts use the prior interval as the baseline, which means that at any moment, it'll auto-resolve even if the regression is still live, if there's no change from the prior interval.

For example, if I've set up a metric alert for 15-min interval, it'll auto-resolve 15 minutes after firing if there's no change from the prior interval. The more accurate baseline is the original interval that triggered the alert.

### Expected Result

The baseline for the metric alert is based on the original data that triggered the alert.

### Actual Result

Recommended fix:

* Use % comparison to trigger the alert
* Use the % comparison to convert to a number-based threshold
* For example - let's say my trigger is 20% change for same time in a week. If on Monday, 3/1 at 9am, I see that my alert triggers b/c count goes from 200 to 220 week over week, on 3/8, if the count is still at 220, I don't auto-resolve....instead, we we'd check against the original value of 200

Dan's notes:

* "resolution_threadhold is already a AlertRule level property. so we add override_resolve_threshold or something to the Incident and when we fill in a value, then we use that instead of the alert rule's threshold "

Places in code to change:
[https://github.com/getsentry/sentry/blob/e1e99b1ce77904a3970c2f180a72ba10724edf8a/src/sentry/incidents/subscription_processor.py#L147-L178]()
[https://github.com/getsentry/sentry/blob/f93d3584763a8d24e038b6885b0c3ca45e59ea63/src/sentry/incidents/logic.py#L124-L133]()

### Product Area

Alerts

### Link

*No response*

### DSN

*No response*

### Version

*No response*

Contributor guide

Open the contributing guide

Research direction

Start by reading the alert handling in src/sentry/incidents/subscription_processor.py around lines 147-178 and the related logic in src/sentry/incidents/logic.py around lines 124-133. Trace how percentage comparisons become thresholds and how resolution is evaluated. Done means percentage-based alerts retain the original triggering baseline and do not auto-resolve while the regression remains unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.