influxdata / influxdata/kapacitor

Configuring alert messages for deadman function

Open
#2,278 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

question
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

I have a batch node combined with a query node that uses a groupBy. This groupBy groups on runtime-name, so the runtime-name appears in the Tag map. Further, I have two string variables that reference that runtime-name Tag. These are the variables (**See template below**):

var start_summary = ‘[PROBLEM] Start summary text for runtime {{ index .Tags “runtime-name” }}’

var end_summary = ‘[FIXED] End summary text for runtime {{ index .Tags “runtime-name” }}’

I added these variables to the Tag map by using the DefaultNode so I can reference them via a Golang template via the construction with the curly braces.

Then, following the batch node is the deadman function. For the deadman function I use the following message (this message refers to the Tags that were created for the variables):

var message = ‘{{ if eq .Level “CRITICAL” }} {{index .Tags “start_summary”}} {{ else if eq .Level “OK” }} {{index .Tags “end_summary”}} {{ end }}’

The deadman function is used to detect if during a certain period of time data were missing. If for a certain amount of time there was no data, and Level gets the value CRITICAL, I want to see the value of variable start_summary, and when the value of Level changes to OK again, I want to see the value of the variable end_summary.
Lets assume that the name of the involved runtime is ‘system.container’.

When Level gets the value CRITICAL, I want to see:

[PROBLEM] Start summary text for runtime system.container

But what I see is:

[PROBLEM] Start summary text for runtime {{ index .Tags “runtime-name” }}

The runtime-Tag in the curly braces doesn’t get evaluated. The same happens when Level goes from CRITICAL to OK again. Then, I want to see the message:

[FIXED] End summary text for runtime system.container

I get:

[FIXED] End summary text for runtime {{ index .Tags “runtime-name” }}

Is it possible to let the deadman function output a message, depending on values of alert data where the messages are in the Tag map and where the messages themselves reference a Tag? Is it possible to fix that the Tag in those messages gets evaluated? Or is a different approach is needed for letting messages depend on values of alert data? What are the possibilities?

**Template:**

var message = ‘{{ if eq .Level “CRITICAL” }} {{index .Tags “start_summary”}} {{ else if eq .Level “OK” }} {{index .Tags “end_summary”}} {{ end }}’

var dayRestriction lambda
var timeRestriction lambda

var qry = ‘SELECT count(*) FROM "’ + db + ‘"."’ + rp + ‘"."’ + measurement + '" WHERE ’ + whereClause

var data = batch
|query(qry)
.period(period)
.cron(qryCronExpr)
.groupBy(‘environment’, ‘runtime-name’)
|default().tag(‘start_summary’, start_summary)
|default().tag(‘end_summary’, end_summary)

var trigger = data
|deadman(threshold, interval, lambda: dayRestriction AND timeRestriction)
.message(message)
.stateChangesOnly()
.details(’’)
.post(‘http://…’)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the deadman function and its .message template handling, using the supplied batch, groupBy, DefaultNode tags, and nested Go-template example as the reproduction. Determine whether nested tag templates are supported and what approach should be documented or changed; done means a clear supported behavior or an agreed fix for CRITICAL and OK messages.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.