influxdata / influxdata/kapacitor
Deadman alert produces errors in derivative node
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
I have created a script that triggers an alert when no point is received for 30s on the ping measurement using the deadman node.
The issue, as shown doing a 'kapacitor show' on the tickscript, is that the derivative node contained in the deadman node produces errors :
var data = stream
|from()
.measurement('ping')
.groupBy('url')
|deadman(0.0, 30s)
.message('Ping: {{ index .Tags "url" }} is {{ if eq .Level "OK" }}up{{ else }}down{{end}}')
.details('{{ if eq .Level "OK" }}{{ index .Tags "url" }} is back online after at least {{ .Duration }} without responding to pings.{{ else }} No ping data received for ' + string(period) + ' from {{ index .Tags "url" }}{{ end }}')
.log('/tmp/deadman.log')
.stateChangesOnly()
.topic('deadman')
DOT:
digraph ping-deadman-adeo {
graph [throughput="0.00 points/s"];
stream0 [avg_exec_time_ns="0s" errors="0" working_cardinality="0" ];
stream0 -> from1 [processed="5080"];
from1 [avg_exec_time_ns="17.127µs" errors="0" working_cardinality="0" ];
from1 -> noop3 [processed="5080"];
noop3 [avg_exec_time_ns="0s" errors="0" working_cardinality="0" ];
stats2 [avg_exec_time_ns="80.236µs" errors="0" working_cardinality="0" ];
stats2 -> derivative4 [processed="1162"];
derivative4 [avg_exec_time_ns="32.133µs" errors="28" working_cardinality="7" ];
derivative4 -> alert5 [processed="1127"];
alert5 [alerts_inhibited="0" alerts_triggered="0" avg_exec_time_ns="80.894µs" crits_triggered="0" errors="0" infos_triggered="0" oks_triggered="0" warns_triggered="0" working_cardinality="7" ];
}
Also, it appears that when grouping by host the alerts trigger fine, whereas when grouping by url like above, no alerts are triggered despite a machine no longer creating points
Contributor guide
Research direction
Start by reproducing the tickscript with `kapacitor show`, focusing on the derivative node inside `deadman(0.0, 30s)` and the difference between grouping by `url` and `host`. Trace the derivative errors and alert behavior; done means the grouped-by-url case produces no derivative errors and triggers alerts when points stop arriving.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100