getsentry / getsentry/sentry

Org based rate limiter for notifications

Open
#68,014 2 comments 0 reactions 0 assignees View on GitHub
Improvement Product Area: Alerts
Dominant language
Python
Stars
44.8k
Forks
4.9k
Avg merge
22h 21m
Merged PRs (30d)
586

Description

Incident link: https://www.notion.so/sentry/874c8e96105e425baca0bed2ba76a8a4

As a followup to a recent incident, we should implement safeguards to ensure that a single org's notifications don't backlog the notifications for all other orgs. The proposal in the linked incident doc suggests adding a rate limit on notifications we can send to a single org.

Requirements:

1. What is the rate limit? (EM to work with Eng on this, but we can set some static ceiling based on [this email task monitor](https://app.datadoghq.com/dashboard/j6i-rfz-67y/queue-status?fullscreen_end_ts=1681367220000&fullscreen_paused=true&fullscreen_start_ts=1681349220000&fullscreen_widget=64099758&tile_focus=64099758&tpl_var_scope%5B0%5D=instance%3Asentry.tasks.email.send_email&from_ts=1681349220000&to_ts=1681367220000&live=false), and combine that with notifications we send to other platforms (Slack, etc)
2. What happens to notifications when they get rate limited?
3. This feature should be turned off for self hosted (need to confirm this)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.