influxdata / influxdata/kapacitor
Watch 3 processes trigger alert if one failed and show dead process
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
I am watching 3 processes via procstat and trigger an alert if one is dead. But I would like to have the name of that dead process in the alert message. How can I define the alert message in a way that I can read the name of the dead process? Maybe the solution is very simple but I am completely new in kapacitor and I couldn't find any hints for that. Below is my tick script
`var db = 'telegraf'
var rp = 'autogen'
var measurement = 'procstat'
var groupBy = []
var whereFilter = lambda: ("process_name" == 'ctdbd' OR "process_name" == 'glusterd' OR "process_name" == 'smbd')
var period = 1m
0s
var name = 'ClusterServices'
var idVar = name
var message = ' {{.ID}} {{.Level}} {{ index .Tags "process_name" }}'
var idTag = 'alertID'
var levelTag = 'level'
var messageField = 'message'
var durationField = 'duration'
var outputDB = 'chronograf'
var outputRP = 'autogen'
var outputMeasurement = 'alerts'
var triggerType = 'deadman'
var threshold = 0.0
var data = stream
|from()
.database(db)
.retentionPolicy(rp)
.measurement(measurement)
.groupBy(groupBy)
.where(whereFilter)
var trigger = data
|deadman(threshold, period)
.message(message)
.id(idVar)
.idTag(idTag)
.levelTag(levelTag)
.messageField(messageField)
.durationField(durationField)
.stateChangesOnly()
.email()
.to('dspisla@dummy.com')
trigger
|eval(lambda: "emitted")
.as('value')
.keep('value', messageField, durationField)
|eval(lambda: float("value"))
.as('value')
.keep()
|influxDBOut()
.create()
.database(outputDB)
.retentionPolicy(outputRP)
.measurement(outputMeasurement)
.tag('alertName', name)
.tag('triggerType', triggerType)
trigger
|httpOut('output')
`
Contributor guide
Research direction
Start with the posted TICK script and its deadman().message(...) configuration; inspect how process_name is represented in the filtered procstat groups and how the email alert renders message templates. Done means an alert for a failed ctdbd, glusterd, or smbd identifies the dead process.
Written by the indexing model from the issue text.
Assessment
- Domain
- observability-sre
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100