Use crons to monitor metric alerts
- Dominant language
- Python
- Stars
- 44.8k
- Forks
- 4.9k
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 624
Description
Metric alerts creates query subscriptions in snuba, and then relies on snuba running these subscriptions and sending updates every `SnubaQuery.resolution` minutes.
One flaw here is that if some kind of bug happened in Snuba that caused some number of subscriptions to break and stop sending, then we wouldn't realise it, since we are just passively ingesting results from Snuba, but don't currently have checks to make sure things are working as expected.
Ideally we'd have some monitoring for this. Crons seems like an ideal candidate - we could create a Cron for every monitor, then when ingesting each result, start an in progress checking when received, and an ok once we've successfully processed. This would give us confidence that metric alerts are working consistently.
There are a couple of downsides here with how crons is built at the moment:
- If we have some kind of general outage, we'll end up with 50k issues (one per cron),which is not ideal
- The Crons UI doesn't handle a large number of crons well - we'd have a huge number of pages of crons to iterate through.
We're discussing a concept of crons being able to monitor an entire system - so the system would be `Metric Alerts`, and then it would have 50k associated cron jobs. We'd then ideally be able to detect system wide outages and only alert once, and also show a more useful ui. Metric alerts could be a good first customer of this product if/when we build it.
Contributor guide
Assessment
This issue has not been assessed yet.