influxdata / influxdata/kapacitor

disable global deadman for some scripts

Open
#2,078 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

Hello,

I have deadman configured globally:
kapacitor.conf:
```toml
[deadman]
# Configure a deadman's switch
# Globally configure deadman's switches on all tasks.
# NOTE: for this to be of use you must also globally configure at least one alerting method.
##global = false
global = true
# Threshold, if globally configured the alert will be triggered if the throughput in points/interval is <= threshold.
threshold = 0.0
# Interval, if globally configured the frequency at which to check the throughput.
interval = "300s"
message = "...."
```
I have a tick script using count():
```batch
var data = stream
|from()
.measurement('xxxxxx')
.groupBy('xxxxxx')
.where(lambda: "xxxx" !~ /xxxxxx/)

data |default()
.field('xxxx', 'xxxx')
.tag('xxxx', '')
|window()
.period(period)
.every(every)
.align()
|count('xxxx')
.as('my_val')

var alert = data
|alert()
...
```

My problem is that it may occur, and this is normal, that I have no value with the ```where``` criteria for long period. Then, count is null, and deadman sends me an alert, and also logs in log file:
```batch
error evaluating expression for level" service=kapacitor task_master=main task=xxxx node=alert5 err="left reference value \"my_val\" is missing value" level=CRITICAL
```
I would like to disable deadman for this script, and also to avoid the logs.
I have tested a lot of things without success.
Is it possible?

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by tracing how the global deadman configuration is applied to tasks and how missing count values reach alert evaluation; done means a task can opt out of the global deadman behavior without producing the reported erroneous log.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.