influxdata / influxdata/kapacitor

Receiving 2 email alerts for an event

Open
#2,458 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

I have setup an alert rule with a check on the cpu usage > 40%
I am receiving 2 emails with different levels (CRITICAL when the threshold is crossed i.e. > 40%) and OK when the usage comes back to normal i.e. < 40%.
What I really want is, receive a single email when the CPU usage crosses 40%. Please help me.

Tick Script generated using chronograf:

var db = 'sample'

var rp = 'default_rp'

var measurement = 'cpu'

var groupBy = []

var whereFilter = lambda: TRUE

var name = 'cpu_test'

var idVar = name

var message = ''

var idTag = 'alertID'

var levelTag = 'level'

var messageField = 'message'

var durationField = 'duration'

var outputDB = 'chronograf'

var outputRP = 'autogen'

var outputMeasurement = 'alerts'

var triggerType = 'threshold'

var crit = 40

var data = stream
|from()
.database(db)
.retentionPolicy(rp)
.measurement(measurement)
.groupBy(groupBy)
.where(whereFilter)
|eval(lambda: "usage_user")
.as('value')

var trigger = data
|alert()
.crit(lambda: "value" > crit)
.stateChangesOnly()
.message(message)
.id(idVar)
.idTag(idTag)
.levelTag(levelTag)
.messageField(messageField)
.durationField(durationField)
.email()

trigger
|eval(lambda: float("value"))
.as('value')
.keep()
|influxDBOut()
.create()
.database(outputDB)
.retentionPolicy(outputRP)
.measurement(outputMeasurement)
.tag('alertName', name)
.tag('triggerType', triggerType)

trigger
|httpOut('output')

I also tried to remove .stateChangesOnly() and modify var whereFilter = lambda: ("usage_user" > 40) but in this case, I just received a single alert when the usage crossed 40%. After sometime it came back to normal. But, the next time when it crossed 40%, there was no alert triggered.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the alert behavior from the provided Kapacitor TICK script, focusing on the alert() configuration, .crit(), and .stateChangesOnly() calls. Compare the generated CRITICAL and OK events with the later behavior when stateChangesOnly is removed; done means alert transitions work consistently on repeated threshold crossings.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.