Percent-change metric alert reports unrealistic deviation
- Dominant language
- Python
- Stars
- 44.8k
- Forks
- 4.9k
- Avg merge
- 21h 10m
- Merged PRs (30d)
- 635
Description
### Environment
SaaS (https://sentry.io/)
### Steps to Reproduce
Percent-change metric alerts are reporting wildly inaccurate deviation values. With a maximum deviation threshold set to 20%, alerts are firing and reporting values like 113% higher compared to the same time one day ago.
However, inspecting the p90(span.duration) chart over the full 24-hour window shows a stable, consistent signal between ~3000–4000ms with no gaps, zero values, or anomalous drops. The dashed comparison baseline tracks nearly identically to the current period. There is no point in the visible data where a ~113% deviation could be computed from the two windows being compared.
### Expected Result
The percent change reported in the alert notification matches the actual deviation visible in the span duration chart for the same time windows.
### Actual Result
The alert fires reporting a 113% increase in p90(span.duration) over the last 30 minutes compared to the same time one day ago.
### Product Area
Alerts
### Link
_No response_
### DSN
_No response_
### Version
_No response_
Contributor guide
Research direction
Reproduce the alert in Sentry SaaS using p90(span.duration), a 20% maximum deviation threshold, and the last 30 minutes compared with the same period one day earlier. Compare the notification's reported percentage with the full 24-hour chart; done when both windows produce matching deviation values.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100