influxdata / influxdata/kapacitor

Rules with exceptions

Open
#1,136 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

Hello!

I got a question and can not find the resolution I would not doubt in.

The situation is following:

Different hosts has different "normal" values of generic metrics, let's say Load Average:
- worker-class hosts LA should always be above "1.0 per cpu"
- storage-class hosts LA should always be relatively low
- database-class hosts LA should be low too but high peaks are possible
- supervisor-class hosts LA is notably higher than "1.0" and that's OK
- etc

Which is the correct way to form LA alert then?

Simple `.warn(lambda: "mean" > 1.0)` doesn't seems suitable cause of different hosts classes. Backup hosts would generate nighly false-positive alerts when backup process occur.
It's also annoying to see tens of alerts when overall load of worker-class hosts is high.

So the questions are:
- What is best practice to separate different alert profiles? Should I just send `load_profile` value for each host or probably include some smart separation logic into TICK scripts?
- Is it possible to somehow unite notable amount of alerts into single meta-alert? Like, not 10 times "web server is overloaded" but "web cluster looks overloaded", which would suppress "childs".

Appreciate any ideas. Thank you for your time.

Contributor guide

Open the contributing guide

Research direction

No file, test, or entry point is identified. Start by reviewing the existing Kapacitor alerting and TICK script guidance, then clarify whether the goal is host-specific alert profiles, grouped meta-alerts, or both. Done would be a documented, agreed approach or a narrowly scoped implementation plan.

Written by the indexing model from the issue text.

Assessment

Domain
observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.