getsentry / getsentry/sentry

Percent-change metric alert reports unrealistic deviation

Open
#113,912 3 comments 0 reactions 0 assignees View on GitHub
Bug Product Area: Alerts Quality Sprint Waiting for: Product Owner
Dominant language
Python
Stars
44.8k
Forks
4.9k
Avg merge
21h 10m
Merged PRs (30d)
635

Description

### Environment

SaaS (https://sentry.io/)

### Steps to Reproduce

Percent-change metric alerts are reporting wildly inaccurate deviation values. With a maximum deviation threshold set to 20%, alerts are firing and reporting values like 113% higher compared to the same time one day ago.

However, inspecting the p90(span.duration) chart over the full 24-hour window shows a stable, consistent signal between ~3000–4000ms with no gaps, zero values, or anomalous drops. The dashed comparison baseline tracks nearly identically to the current period. There is no point in the visible data where a ~113% deviation could be computed from the two windows being compared.

### Expected Result

The percent change reported in the alert notification matches the actual deviation visible in the span duration chart for the same time windows.

### Actual Result

The alert fires reporting a 113% increase in p90(span.duration) over the last 30 minutes compared to the same time one day ago.

### Product Area

Alerts

### Link

_No response_

### DSN

_No response_

### Version

_No response_

Contributor guide

Open the contributing guide

Research direction

Reproduce the alert in Sentry SaaS using p90(span.duration), a 20% maximum deviation threshold, and the last 30 minutes compared with the same period one day earlier. Compare the notification's reported percentage with the full 24-hour chart; done when both windows produce matching deviation values.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.