influxdata / influxdata/kapacitor

Problem with Deadman function

Open
#2,419 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

Good morning all! I need your help with this issue.
For a while i've been using this Deadman code you imply it's the same as the function |deadman():
```
data
|stats(10s)
.align()
|derivative('emitted')
.unit(10s)
.nonNegative()
|alert()
.id('node \'stream0\' in task \'{{ .TaskName }}\'')
.message('{{ .ID }} is {{ if eq .Level "OK" }}alive{{ else }}dead{{ end }}: {{ index .Fields "emitted" | printf "%0.3f" }} points/10s.')
.crit(lambda: "emitted" <= 100.0)
```
I've been doing a test using booth alarms and the stats+derivative vs deadman function . We have found that stats+derivative method has false positives, deadman has not.
It's true that we are using stats+derivative in batch mode while we are using deadman with stream mode. But it shouldn't change anything.
Can you check it? Please, we need our alarms to be reliable, and in this moment we are not having any confidence because of all this false positive.
Thank you for your support I expect an answer ASAP (editado)

Contributor guide

Open the contributing guide

Research direction

Start by comparing the provided stats(10s)+derivative alert in batch mode with the deadman alert in stream mode, focusing on the reported false positives. Reproduce the behavior with both alarms and establish why their reliability differs; done means the discrepancy is explained and the false positives are addressed.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.