influxdata / influxdata/kapacitor

Combining alert state for multiple alerts

Open
#762 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

I am trying to get alerts with specific time windows with multiple batch nodes and same alert ID in a script:

```
batch
|query('''
SELECT .......
''')
.align()
.period(1m)
.offset(1m)
.every(1m)
|where(lambda: ( some time conditions, say 1 PM to 4 PM)
|stats(1m)
|derivative('emitted')
.unit(1m)
.as('xyz')
|alert()
.crit( lambda: "xyz condition")
.stateChangesOnly()
```

The time conditions are disjoint.
I am getting two CRITICAL alerts, at the time when time condition changes, i.e, at 4 PM, one from first node and one from second. (Second one is expected)
Now I need some way to combine these alerts, because the second alert gets state change in some time, but when the time window is switched back to that of first alert (at 1 PM), new alert does not get triggered because it is still in same CRITICAL state.

Is it possible to combine this state change?

Contributor guide

Open the contributing guide

Research direction

Start by tracing the alert() and stateChangesOnly() stages in Kapacitor's alert-state handling, focusing on how separate batch nodes and disjoint time windows maintain state. Done means a defined way to combine those alert states so switching back to the first window produces the needed state change, with behavior verified for both nodes.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.