Org based rate limiter for notifications
- Dominant language
- Python
- Stars
- 44.8k
- Forks
- 4.9k
- Avg merge
- 22h 21m
- Merged PRs (30d)
- 586
Description
Incident link: https://www.notion.so/sentry/874c8e96105e425baca0bed2ba76a8a4
As a followup to a recent incident, we should implement safeguards to ensure that a single org's notifications don't backlog the notifications for all other orgs. The proposal in the linked incident doc suggests adding a rate limit on notifications we can send to a single org.
Requirements:
1. What is the rate limit? (EM to work with Eng on this, but we can set some static ceiling based on [this email task monitor](https://app.datadoghq.com/dashboard/j6i-rfz-67y/queue-status?fullscreen_end_ts=1681367220000&fullscreen_paused=true&fullscreen_start_ts=1681349220000&fullscreen_widget=64099758&tile_focus=64099758&tpl_var_scope%5B0%5D=instance%3Asentry.tasks.email.send_email&from_ts=1681349220000&to_ts=1681367220000&live=false), and combine that with notifications we send to other platforms (Slack, etc)
2. What happens to notifications when they get rate limited?
3. This feature should be turned off for self hosted (need to confirm this)
Contributor guide
Assessment
This issue has not been assessed yet.