getsentry / getsentry/sentry

Ratio Based Alert Support

Open
#106,102 3 comments 1 reaction 0 assignees View on GitHub
Feature Product Area: Alerts
Dominant language
Python
Stars
44.8k
Forks
4.9k
Avg merge
21h 10m
Merged PRs (30d)
635

Description

### Problem Statement

It would be really useful to configure alerts that are based on a ratio, such as a Span equation that uses the division operation.
This would enable alerting not just on the existence of some critical behavior, but on whether the percentage of traffic experiencing that behavior is statistically significant compared to our overall traffic.

For example, an application might block unauthorized traffic and return a 401 response. Spikes of 401 wouldn't be unusual, so an anomaly detection approach wouldn't quite fit. Despite that, we might still want to know when a majority of traffic being served is just 401s - it tells us that a majority of our cost-to-serve is being wasted on these unauthorized requests, and it might be time to add some additional service protections (ex. IP blocks for nefarious traffic).

We can add a dashboard view that shows what percentage of traffic is being served 401s with something like this:
`count_if(span.status_code, equals, 401) / count(span.duration)`
Unfortunately we can't alert on that type of query.

### Solution Brainstorm

It would be great to support the same Span Equation query in an Alert that is supported in the dashboard.
This could be a new Alert Type, and would accept the same Span Equation arguments that are currently supported in the trace visualization view (and dashboard views)

### Product Area

Alerts

Contributor guide

Open the contributing guide

Research direction

Review the existing Span Equation support in the trace visualization and dashboard views, then compare how alert types evaluate queries and thresholds. Determine how the same equation arguments could be accepted by an alert, and consider the ratio example in the issue as the completion case. Done means the alert can evaluate the supported Span Equation and trigger according to its configured threshold.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.