influxdata / influxdata/kapacitor
Deadman resets critical when using lambda filter
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
I am using deadman to test my cron jobs. i-e cron jobs push a metric to influx when they run and we send an alert if we notice that a point that should have been there is missing
kapacitor template
```
var period = 1d
var every = 10m
var where string
var lambda_filter lambda
var cron = batch
|query('''SELECT count("build_number") AS stat FROM "jenkins"."autogen"."jenkins_data"'''+''' '''+where)
.groupBy(time(every), 'project_name')
.period(period)
.every(every)
|log()
|deadman(1.0, every,lambda: lambda_filter)
.id('deadman/{{ index .Tags "project_name" }}')
.message('{{ .Level }} - Cron {{ index .Tags "project_name" }} {{ if ne .Level "OK" }}has not run since '+string(period)+'!{{ else }} is back online.{{ end }}')
.stateChangesOnly()
.slack()
.channel('#test-kapacitor')
```
var
```
{
"where": {"type" : "string", "value" : "where project_name =~ /nightly/" },
"every": {"type" : "duration", "value" : "1h" },
"period": {"type" : "duration", "value" : "6h" },
"lambda_filter": {"type" : "lambda", "value" : "hour(\"time\") >= 7 AND hour(\"time\") <= 8" }
}
```
We get critical notifications at 7 & 8 for different cron jobs but then at 9 we receive OK notifications for all which is wrong.
slack logs:
[9:00 AM]
CRITICAL - Cron nightly_admin_generate_ramp_up_tasks has not run since 6h!
[10:00]
CRITICAL - Cron nightly_admin_lead_rotation_warnings has not run since 6h!
[11:00 AM]
OK - Cron nightly_admin_generate_ramp_up_tasks is back online.
[11:00]
OK - Cron nightly_admin_lead_rotation_warnings is back online.
Data in influx is in utc and we are utc+2 that's why alert at 9am and 10am in slack
Contributor guide
Research direction
Start by reproducing the supplied Kapacitor template with the deadman lambda filter, the hourly period/every settings, and UTC data against the reported UTC+2 notification times. Trace how the lambda_filter affects deadman state transitions for each project, especially the transition from critical to OK. Done means the behavior matches the expected per-project alerts and does not reset all alerts at the following hour.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100