influxdata / influxdata/kapacitor
[ Feature Request ] Adding "INIT" state to alert nodes
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
I have a use case where on startup of kapacitor, I would like to see alert nodes start in an "INIT"/"UNKNOWN" state. This would force a transition to one of the other states when the alert node emits. Guess you could also consider this a mechanism similar to ```.stateChangesOnly()``` but more like ```.afterFirstEmitStateChangesOnly()```
My specific use case is to have the state of an alert node fed back into InfluxDB, so I can track task states over time as well as to build a dashboard of the current state (See tick snippet below). If a specific task starts in the OK state and the data used to trigger alert node state changes is "OK" then I will never get a state change. I need to wait for a real set of conditions to occur to transition from OK to a higher level.
So unless a task alert node has gone into a non OK state, my current state would be unknown.
I would propose some type of way to have alert nodes start in an INIT state, via configuration or perhaps as a property in the node itself.
```
.initialState("INIT")
```
In this case specifically I would expect that on start of kapacitor any state loaded from disk would be ignored and the transition from INIT to OK would be based on the alert node **.warn** **.crit** properties.
```
data
|alert()
.stateChangesOnly()
.all() //Check that all points in the window against the lambda
.warn(lambda: "jvm_memory_pressure_maximum" > elasticsearch_memory_pressure_warn_level)
.crit(lambda: "jvm_memory_pressure_maximum" > elasticsearch_memory_pressure_critical_level)
.message(string(alert_header) + '
Cluster \'' + string(elasticsearch_domain_name) + '\'
JVM memory pressure is at {{ index .Fields "jvm_memory_pressure_maximum" | printf "%02.f"}}%
Expecting < {{ if eq .Level "CRITICAL" }}' + string(elasticsearch_memory_pressure_critical_level) + '{{ else }}' + string(elasticsearch_memory_pressure_warn_level) + '{{ end }}%
<' + string(elasticsearch_memory_pressure_graph_url) + '|Grafana>')
.slack()
.channel(slack_channel)
.levelField('level')
|eval(lambda: if("level" == 'CRITICAL', 2, if("level" == 'WARNING' , 1, 0)))
.as('state')
.keep()
|influxDBOut()
.database(ops_alerts_db)
.retentionPolicy(ops_alerts_rp)
.measurement(taskname)
.tag('stack', stack)
.tag('deployment', deployment)
```
Contributor guide
Research direction
Start with the alert() node's state initialization and the handling of state loaded from disk. Define how an INIT state should transition through the .warn and .crit conditions, and verify that the requested behavior supports state tracking through influxDBOut().
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100