influxdata / influxdata/kapacitor

[ Feature Request ] Adding "INIT" state to alert nodes

Open
#1,297 0 comments 4 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

I have a use case where on startup of kapacitor, I would like to see alert nodes start in an "INIT"/"UNKNOWN" state. This would force a transition to one of the other states when the alert node emits. Guess you could also consider this a mechanism similar to ```.stateChangesOnly()``` but more like ```.afterFirstEmitStateChangesOnly()```

My specific use case is to have the state of an alert node fed back into InfluxDB, so I can track task states over time as well as to build a dashboard of the current state (See tick snippet below). If a specific task starts in the OK state and the data used to trigger alert node state changes is "OK" then I will never get a state change. I need to wait for a real set of conditions to occur to transition from OK to a higher level.
So unless a task alert node has gone into a non OK state, my current state would be unknown.

I would propose some type of way to have alert nodes start in an INIT state, via configuration or perhaps as a property in the node itself.
```
.initialState("INIT")
```

In this case specifically I would expect that on start of kapacitor any state loaded from disk would be ignored and the transition from INIT to OK would be based on the alert node **.warn** **.crit** properties.

```
data
|alert()
.stateChangesOnly()
.all() //Check that all points in the window against the lambda
.warn(lambda: "jvm_memory_pressure_maximum" > elasticsearch_memory_pressure_warn_level)
.crit(lambda: "jvm_memory_pressure_maximum" > elasticsearch_memory_pressure_critical_level)
.message(string(alert_header) + '
Cluster \'' + string(elasticsearch_domain_name) + '\'
JVM memory pressure is at {{ index .Fields "jvm_memory_pressure_maximum" | printf "%02.f"}}%
Expecting < {{ if eq .Level "CRITICAL" }}' + string(elasticsearch_memory_pressure_critical_level) + '{{ else }}' + string(elasticsearch_memory_pressure_warn_level) + '{{ end }}%
<' + string(elasticsearch_memory_pressure_graph_url) + '|Grafana>')
.slack()
.channel(slack_channel)
.levelField('level')
|eval(lambda: if("level" == 'CRITICAL', 2, if("level" == 'WARNING' , 1, 0)))
.as('state')
.keep()
|influxDBOut()
.database(ops_alerts_db)
.retentionPolicy(ops_alerts_rp)
.measurement(taskname)
.tag('stack', stack)
.tag('deployment', deployment)
```

Contributor guide

Open the contributing guide

Research direction

Start with the alert() node's state initialization and the handling of state loaded from disk. Define how an INIT state should transition through the .warn and .crit conditions, and verify that the requested behavior supports state tracking through influxDBOut().

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.