influxdata / influxdata/kapacitor

Deadman alert produces errors in derivative node

Open
#1,984 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

I have created a script that triggers an alert when no point is received for 30s on the ping measurement using the deadman node.
The issue, as shown doing a 'kapacitor show' on the tickscript, is that the derivative node contained in the deadman node produces errors :

var data = stream
|from()
.measurement('ping')
.groupBy('url')
|deadman(0.0, 30s)
.message('Ping: {{ index .Tags "url" }} is {{ if eq .Level "OK" }}up{{ else }}down{{end}}')
.details('{{ if eq .Level "OK" }}{{ index .Tags "url" }} is back online after at least {{ .Duration }} without responding to pings.{{ else }} No ping data received for ' + string(period) + ' from {{ index .Tags "url" }}{{ end }}')
.log('/tmp/deadman.log')
.stateChangesOnly()
.topic('deadman')

DOT:
digraph ping-deadman-adeo {
graph [throughput="0.00 points/s"];

stream0 [avg_exec_time_ns="0s" errors="0" working_cardinality="0" ];
stream0 -> from1 [processed="5080"];

from1 [avg_exec_time_ns="17.127µs" errors="0" working_cardinality="0" ];
from1 -> noop3 [processed="5080"];

noop3 [avg_exec_time_ns="0s" errors="0" working_cardinality="0" ];

stats2 [avg_exec_time_ns="80.236µs" errors="0" working_cardinality="0" ];
stats2 -> derivative4 [processed="1162"];

derivative4 [avg_exec_time_ns="32.133µs" errors="28" working_cardinality="7" ];
derivative4 -> alert5 [processed="1127"];

alert5 [alerts_inhibited="0" alerts_triggered="0" avg_exec_time_ns="80.894µs" crits_triggered="0" errors="0" infos_triggered="0" oks_triggered="0" warns_triggered="0" working_cardinality="7" ];
}

Also, it appears that when grouping by host the alerts trigger fine, whereas when grouping by url like above, no alerts are triggered despite a machine no longer creating points

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the tickscript with `kapacitor show`, focusing on the derivative node inside `deadman(0.0, 30s)` and the difference between grouping by `url` and `host`. Trace the derivative errors and alert behavior; done means the grouped-by-url case produces no derivative errors and triggers alerts when points stop arriving.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.