Ratio Based Alert Support
- Dominant language
- Python
- Stars
- 44.8k
- Forks
- 4.9k
- Avg merge
- 21h 10m
- Merged PRs (30d)
- 635
Description
### Problem Statement
It would be really useful to configure alerts that are based on a ratio, such as a Span equation that uses the division operation.
This would enable alerting not just on the existence of some critical behavior, but on whether the percentage of traffic experiencing that behavior is statistically significant compared to our overall traffic.
For example, an application might block unauthorized traffic and return a 401 response. Spikes of 401 wouldn't be unusual, so an anomaly detection approach wouldn't quite fit. Despite that, we might still want to know when a majority of traffic being served is just 401s - it tells us that a majority of our cost-to-serve is being wasted on these unauthorized requests, and it might be time to add some additional service protections (ex. IP blocks for nefarious traffic).
We can add a dashboard view that shows what percentage of traffic is being served 401s with something like this:
`count_if(span.status_code, equals, 401) / count(span.duration)`
Unfortunately we can't alert on that type of query.
### Solution Brainstorm
It would be great to support the same Span Equation query in an Alert that is supported in the dashboard.
This could be a new Alert Type, and would accept the same Span Equation arguments that are currently supported in the trace visualization view (and dashboard views)
### Product Area
Alerts
Contributor guide
Research direction
Review the existing Span Equation support in the trace visualization and dashboard views, then compare how alert types evaluate queries and thresholds. Determine how the same equation arguments could be accepted by an alert, and consider the ratio example in the issue as the completion case. Done means the alert can evaluate the supported Span Equation and trigger according to its configured threshold.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100