influxdata / influxdata/kapacitor

Deadman resets critical when using lambda filter

Open
#1,585 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

I am using deadman to test my cron jobs. i-e cron jobs push a metric to influx when they run and we send an alert if we notice that a point that should have been there is missing

kapacitor template
```
var period = 1d
var every = 10m
var where string
var lambda_filter lambda

var cron = batch
|query('''SELECT count("build_number") AS stat FROM "jenkins"."autogen"."jenkins_data"'''+''' '''+where)
.groupBy(time(every), 'project_name')
.period(period)
.every(every)
|log()
|deadman(1.0, every,lambda: lambda_filter)
.id('deadman/{{ index .Tags "project_name" }}')
.message('{{ .Level }} - Cron {{ index .Tags "project_name" }} {{ if ne .Level "OK" }}has not run since '+string(period)+'!{{ else }} is back online.{{ end }}')
.stateChangesOnly()
.slack()
.channel('#test-kapacitor')
```

var
```
{
"where": {"type" : "string", "value" : "where project_name =~ /nightly/" },
"every": {"type" : "duration", "value" : "1h" },
"period": {"type" : "duration", "value" : "6h" },
"lambda_filter": {"type" : "lambda", "value" : "hour(\"time\") >= 7 AND hour(\"time\") <= 8" }
}
```
We get critical notifications at 7 & 8 for different cron jobs but then at 9 we receive OK notifications for all which is wrong.

slack logs:

[9:00 AM]
CRITICAL - Cron nightly_admin_generate_ramp_up_tasks has not run since 6h!
[10:00]
CRITICAL - Cron nightly_admin_lead_rotation_warnings has not run since 6h!
[11:00 AM]
OK - Cron nightly_admin_generate_ramp_up_tasks is back online.
[11:00]
OK - Cron nightly_admin_lead_rotation_warnings is back online.

Data in influx is in utc and we are utc+2 that's why alert at 9am and 10am in slack

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the supplied Kapacitor template with the deadman lambda filter, the hourly period/every settings, and UTC data against the reported UTC+2 notification times. Trace how the lambda_filter affects deadman state transitions for each project, especially the transition from critical to OK. Done means the behavior matches the expected per-project alerts and does not reset all alerts at the following hour.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.