influxdata / influxdata/kapacitor
Configuring alert messages for deadman function
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
I have a batch node combined with a query node that uses a groupBy. This groupBy groups on runtime-name, so the runtime-name appears in the Tag map. Further, I have two string variables that reference that runtime-name Tag. These are the variables (**See template below**):
var start_summary = ‘[PROBLEM] Start summary text for runtime {{ index .Tags “runtime-name” }}’
var end_summary = ‘[FIXED] End summary text for runtime {{ index .Tags “runtime-name” }}’
I added these variables to the Tag map by using the DefaultNode so I can reference them via a Golang template via the construction with the curly braces.
Then, following the batch node is the deadman function. For the deadman function I use the following message (this message refers to the Tags that were created for the variables):
var message = ‘{{ if eq .Level “CRITICAL” }} {{index .Tags “start_summary”}} {{ else if eq .Level “OK” }} {{index .Tags “end_summary”}} {{ end }}’
The deadman function is used to detect if during a certain period of time data were missing. If for a certain amount of time there was no data, and Level gets the value CRITICAL, I want to see the value of variable start_summary, and when the value of Level changes to OK again, I want to see the value of the variable end_summary.
Lets assume that the name of the involved runtime is ‘system.container’.
When Level gets the value CRITICAL, I want to see:
[PROBLEM] Start summary text for runtime system.container
But what I see is:
[PROBLEM] Start summary text for runtime {{ index .Tags “runtime-name” }}
The runtime-Tag in the curly braces doesn’t get evaluated. The same happens when Level goes from CRITICAL to OK again. Then, I want to see the message:
[FIXED] End summary text for runtime system.container
I get:
[FIXED] End summary text for runtime {{ index .Tags “runtime-name” }}
Is it possible to let the deadman function output a message, depending on values of alert data where the messages are in the Tag map and where the messages themselves reference a Tag? Is it possible to fix that the Tag in those messages gets evaluated? Or is a different approach is needed for letting messages depend on values of alert data? What are the possibilities?
**Template:**
var message = ‘{{ if eq .Level “CRITICAL” }} {{index .Tags “start_summary”}} {{ else if eq .Level “OK” }} {{index .Tags “end_summary”}} {{ end }}’
var dayRestriction lambda
var timeRestriction lambda
var qry = ‘SELECT count(*) FROM "’ + db + ‘"."’ + rp + ‘"."’ + measurement + '" WHERE ’ + whereClause
var data = batch
|query(qry)
.period(period)
.cron(qryCronExpr)
.groupBy(‘environment’, ‘runtime-name’)
|default().tag(‘start_summary’, start_summary)
|default().tag(‘end_summary’, end_summary)
var trigger = data
|deadman(threshold, interval, lambda: dayRestriction AND timeRestriction)
.message(message)
.stateChangesOnly()
.details(’’)
.post(‘http://…’)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the deadman function and its .message template handling, using the supplied batch, groupBy, DefaultNode tags, and nested Go-template example as the reproduction. Determine whether nested tag templates are supported and what approach should be documented or changed; done means a clear supported behavior or an agreed fix for CRITICAL and OK messages.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100