kiwix / kiwix/operations

Better grafana to Slack alerts

Open
#153 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
15
Forks
5
PR merge metrics
No merged PRs in 30d

Description

With grafana, we now have access to alerts on HTTP 50x errors (on for all 50x) and one for 500 on all our containers. It's a good thing but the way we get those alerts is not ideal:

- we get a message after a varying delay of up to 5mn
- we get a _resolved_ message once the grafana condition for alert is gone.
- the messages are very long and useless, with just the namespace burried inside the extended version of the message

The most annoying to me is the resolve message that creates noise. The fact that I have to switch context to go to slack, select channel and click on the message to find out about the namespace (and infer the culprit container) may look like nothing to care about but it's actually a huge blocker to react on issues.

Uptime robot alerts very clear and useful in comparison. From the notification preview I already know what's going on an can decide whether it requires immediate action or not.

With grafana, I can't so I mostly decide that I'll look at it when I get a chance…

Screenshot 2023-12-16 at 14 30 24

---

Screenshot 2023-12-16 at 14 30 41

---

Screenshot 2023-12-16 at 14 30 51

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the Grafana alert configuration and Slack notification behavior described in the issue, using the screenshots and the Uptime Robot alerts as comparison points. Identify how namespace, container, delay, and resolved-message handling are currently represented. Done means alerts provide actionable context in the preview, with unnecessary resolved-message noise addressed.

Written by the indexing model from the issue text.

Assessment

Tech stack
grafana
Domain
observability-sre
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.