influxdata / influxdata/kapacitor
Problem with Deadman function
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
Good morning all! I need your help with this issue.
For a while i've been using this Deadman code you imply it's the same as the function |deadman():
```
data
|stats(10s)
.align()
|derivative('emitted')
.unit(10s)
.nonNegative()
|alert()
.id('node \'stream0\' in task \'{{ .TaskName }}\'')
.message('{{ .ID }} is {{ if eq .Level "OK" }}alive{{ else }}dead{{ end }}: {{ index .Fields "emitted" | printf "%0.3f" }} points/10s.')
.crit(lambda: "emitted" <= 100.0)
```
I've been doing a test using booth alarms and the stats+derivative vs deadman function . We have found that stats+derivative method has false positives, deadman has not.
It's true that we are using stats+derivative in batch mode while we are using deadman with stream mode. But it shouldn't change anything.
Can you check it? Please, we need our alarms to be reliable, and in this moment we are not having any confidence because of all this false positive.
Thank you for your support I expect an answer ASAP (editado)
Contributor guide
Research direction
Start by comparing the provided stats(10s)+derivative alert in batch mode with the deadman alert in stream mode, focusing on the reported false positives. Reproduce the behavior with both alarms and establish why their reliability differs; done means the discrepancy is explained and the false positives are addressed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100