Metric Monitors: Detect ingestion delays
- Dominant language
- Python
- Stars
- 44.8k
- Forks
- 4.9k
- Avg merge
- 22h 21m
- Merged PRs (30d)
- 586
Description
### Problem Statement
Similar to https://github.com/getsentry/sentry/issues/79328
Challenge:
> Throughput monitoring works well for crons to duck alerting because we have a well known traffic pattern. If we suddenly start receiving less cron check-ins, we can almost be 100% positive it's an internal infrastructure problem.
This is not necessarily the case for other data sets (errors, metrics, logs, spans etc.)
### Solution Brainstorm
We could potentially use the same signals from crons to stop other occurrences however ingestion delays or outages are not always equally distributed
Similar approaches with respect to various forms of telemetry or the ability to mute moinitors/alerts during backfill periods or known issues would result in fewer false positive incidents/pages for customers that rely on these signals
### Product Area
Alerts
Contributor guide
Research direction
Start by reading the related issue #79328 and the discussion in this issue. Clarify which telemetry types, ingestion-delay signals, and mute or backfill behaviors are in scope before identifying implementation files and tests; done should mean an agreed, testable design for reducing false alerts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100