influxdata / influxdata/kapacitor
Rules with exceptions
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
Hello!
I got a question and can not find the resolution I would not doubt in.
The situation is following:
Different hosts has different "normal" values of generic metrics, let's say Load Average:
- worker-class hosts LA should always be above "1.0 per cpu"
- storage-class hosts LA should always be relatively low
- database-class hosts LA should be low too but high peaks are possible
- supervisor-class hosts LA is notably higher than "1.0" and that's OK
- etc
Which is the correct way to form LA alert then?
Simple `.warn(lambda: "mean" > 1.0)` doesn't seems suitable cause of different hosts classes. Backup hosts would generate nighly false-positive alerts when backup process occur.
It's also annoying to see tens of alerts when overall load of worker-class hosts is high.
So the questions are:
- What is best practice to separate different alert profiles? Should I just send `load_profile` value for each host or probably include some smart separation logic into TICK scripts?
- Is it possible to somehow unite notable amount of alerts into single meta-alert? Like, not 10 times "web server is overloaded" but "web cluster looks overloaded", which would suppress "childs".
Appreciate any ideas. Thank you for your time.
Contributor guide
Research direction
No file, test, or entry point is identified. Start by reviewing the existing Kapacitor alerting and TICK script guidance, then clarify whether the goal is host-specific alert profiles, grouped meta-alerts, or both. Done would be a documented, agreed approach or a narrowly scoped implementation plan.
Written by the indexing model from the issue text.
Assessment
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100