influxdata / influxdata/kapacitor

Watch 3 processes trigger alert if one failed and show dead process

Open
#2,024 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

I am watching 3 processes via procstat and trigger an alert if one is dead. But I would like to have the name of that dead process in the alert message. How can I define the alert message in a way that I can read the name of the dead process? Maybe the solution is very simple but I am completely new in kapacitor and I couldn't find any hints for that. Below is my tick script

`var db = 'telegraf'

var rp = 'autogen'

var measurement = 'procstat'

var groupBy = []

var whereFilter = lambda: ("process_name" == 'ctdbd' OR "process_name" == 'glusterd' OR "process_name" == 'smbd')

var period = 1m

0s

var name = 'ClusterServices'

var idVar = name

var message = ' {{.ID}} {{.Level}} {{ index .Tags "process_name" }}'

var idTag = 'alertID'

var levelTag = 'level'

var messageField = 'message'

var durationField = 'duration'

var outputDB = 'chronograf'

var outputRP = 'autogen'

var outputMeasurement = 'alerts'

var triggerType = 'deadman'

var threshold = 0.0

var data = stream
|from()
.database(db)
.retentionPolicy(rp)
.measurement(measurement)
.groupBy(groupBy)
.where(whereFilter)

var trigger = data
|deadman(threshold, period)
.message(message)
.id(idVar)
.idTag(idTag)
.levelTag(levelTag)
.messageField(messageField)
.durationField(durationField)
.stateChangesOnly()
.email()
.to('dspisla@dummy.com')

trigger
|eval(lambda: "emitted")
.as('value')
.keep('value', messageField, durationField)
|eval(lambda: float("value"))
.as('value')
.keep()
|influxDBOut()
.create()
.database(outputDB)
.retentionPolicy(outputRP)
.measurement(outputMeasurement)
.tag('alertName', name)
.tag('triggerType', triggerType)

trigger
|httpOut('output')
`

Contributor guide

Open the contributing guide

Research direction

Start with the posted TICK script and its deadman().message(...) configuration; inspect how process_name is represented in the filtered procstat groups and how the email alert renders message templates. Done means an alert for a failed ctdbd, glusterd, or smbd identifies the dead process.

Written by the indexing model from the issue text.

Assessment

Domain
observability-sre
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.