Better grafana to Slack alerts
- Dominant language
- Python
- Stars
- 15
- Forks
- 5
- PR merge metrics
- No merged PRs in 30d
Description
With grafana, we now have access to alerts on HTTP 50x errors (on for all 50x) and one for 500 on all our containers. It's a good thing but the way we get those alerts is not ideal:
- we get a message after a varying delay of up to 5mn
- we get a _resolved_ message once the grafana condition for alert is gone.
- the messages are very long and useless, with just the namespace burried inside the extended version of the message
The most annoying to me is the resolve message that creates noise. The fact that I have to switch context to go to slack, select channel and click on the message to find out about the namespace (and infer the culprit container) may look like nothing to care about but it's actually a huge blocker to react on issues.
Uptime robot alerts very clear and useful in comparison. From the notification preview I already know what's going on an can decide whether it requires immediate action or not.
With grafana, I can't so I mostly decide that I'll look at it when I get a chance…
---
---
Contributor guide
Research direction
Start by reviewing the Grafana alert configuration and Slack notification behavior described in the issue, using the screenshots and the Uptime Robot alerts as comparison points. Identify how namespace, container, delay, and resolved-message handling are currently represented. Done means alerts provide actionable context in the preview, with unnecessary resolved-message noise addressed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grafana
- Domain
- observability-sre
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100