influxdata / influxdata/kapacitor

OK events are not generating even on matching condition on stateDuration. Data is in stream mode, and using window() node

Open
#2,731 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

var db = 'telegraf'
var rp = 'autogen'
var measurement = 'cpu'
var groupBy = ['host', 'gms-rule']
var period = 5m
var every = 1m
var whereFilter = lambda: isPresent("gms-rule") AND "cpu" == 'cpu-total' AND isPresent("usage_pct") AND "host" == 'venu-test'
var data = stream
|from()
.database(db)
.retentionPolicy(rp)
.measurement(measurement)
.groupBy(groupBy)
.where(whereFilter)
|window()
.period(period)
.every(every)
.align()
|eval(lambda: "usage_pct")
.as('value')
|httpOut('stream')

var trigger = custom
|stateDuration(lambda: "warningEnabled" AND "value" >= "warningThreshold" AND "enable")
.as('actual_warn_duration')
|stateDuration(lambda: "criticalEnabled" AND "value" >= "criticalThreshold" AND "enable")
.as('actual_crit_duration')
|stateDuration(lambda: "value" < "warningThreshold" OR "value" < "criticalThreshold" AND global_ok_stateDuration_enabled)
.as('ok_duration_counter')
|httpOut('trigger')
|log()
|alert()
.warn(lambda: "actual_warn_duration" >= "warningStateDuration")
.crit(lambda: "actual_crit_duration" >= "criticalStateDuration")
// .message(message)
.critReset(lambda: "ok_duration_counter" > "okstateDuration")
.warnReset(lambda: "ok_duration_counter" > "okstateDuration")
.id(idVar)
.levelTag(levelTag)
.messageField(messageField)
.stateChangesOnly()
.log('/apps/helios/influx-ent/logs/kapacitor/a-test-ok-1.log')

With this tick script, i am getting a warning or critical alert, but never doing a reset (no OK events are coming) even the critReset or warnReset conditions are becoming true.

Is this a kind of bug? or reset will not happen when i use a window() node?

If i am not using the window() node, its working as expected. Getting crit/warn alerts, and resetting back to OK on a matching condition.

Contributor guide

Open the contributing guide

Research direction

Start with the supplied TICKscript and compare the alert pipeline with and without the window() node. Exercise the critReset and warnReset conditions while inspecting the stateDuration and alert() stages. Done means determining whether window() changes reset behavior and documenting or correcting the reproducible failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.