influxdata / influxdata/kapacitor
Alert Node failing intermittently using Batch script
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
I am using a Batch script to try and create a simple alert based on disk utilization. The below script functions part of the time and I have yet to discover exactly why it would fail intermittently. Between attempts I define/disable/enable the script. Clearly the data is present inside InfluxDB, and the Mean operation is run succesfully.
- Telegraf is used as Agent to send data into InfluxDB
- Kapacitor is configured globally using `state-changes-only = false`
- Kapacitor v1.4.0
- InfluxDB v1.4.3
- Telegraf v1.5.2-1
## Remote host Telegraf configuration ##
```[[inputs.disk]]
## By default, telegraf gather stats for all mountpoints.
## Setting mountpoints will restrict the stats to the specified mountpoints.
mount_points = ["/"]
```
## Script Configuration ##
```
batch
|query('''SELECT "used" FROM "vpcx_central_monitoring"."rp_weeks_12".disk WHERE ("host" = 'vpc1-az1a-msql-track01')''')
.period(1m)
.every(1m)
.fill(0.0)
|log()
|mean('used')
.as('used')
|log()
|alert()
.warn(lambda: "used" > 1.0)
.log('/var/log/kapacitor/vm-sys.log')
.message('{{ .Level }} for {{ .ID }}')
|log()
```
## Output from kapacitor.log ##
```ts=2018-03-12T00:05:21.341+01:00 lvl=info msg="begin batch" service=kapacitor task_master=main task=kapacitor-task-roll-avg-vpc1-az1a-msql-track01-vm-sys-disk-used node=log2 prefix= name=disk group= time=2018-03-12T00:05:21.327512952+01:00
ts=2018-03-12T00:05:21.342+01:00 lvl=info msg="batch point" service=kapacitor task_master=main task=kapacitor-task-roll-avg-vpc1-az1a-msql-track01-vm-sys-disk-used node=log2 prefix= name=disk group= field_used=16671760384 time=2018-03-11T23:04:30Z
ts=2018-03-12T00:05:21.342+01:00 lvl=info msg="batch point" service=kapacitor task_master=main task=kapacitor-task-roll-avg-vpc1-az1a-msql-track01-vm-sys-disk-used node=log2 prefix= name=disk group= field_used=16671764480 time=2018-03-11T23:04:40Z
ts=2018-03-12T00:05:21.342+01:00 lvl=info msg="batch point" service=kapacitor task_master=main task=kapacitor-task-roll-avg-vpc1-az1a-msql-track01-vm-sys-disk-used node=log2 prefix= name=disk group= field_used=16671764480 time=2018-03-11T23:04:50Z
ts=2018-03-12T00:05:21.343+01:00 lvl=info msg="batch point" service=kapacitor task_master=main task=kapacitor-task-roll-avg-vpc1-az1a-msql-track01-vm-sys-disk-used node=log2 prefix= name=disk group= field_used=16671768576 time=2018-03-11T23:05:00Z
ts=2018-03-12T00:05:21.343+01:00 lvl=info msg="batch point" service=kapacitor task_master=main task=kapacitor-task-roll-avg-vpc1-az1a-msql-track01-vm-sys-disk-used node=log2 prefix= name=disk group= field_used=16671768576 time=2018-03-11T23:05:10Z
ts=2018-03-12T00:05:21.344+01:00 lvl=info msg="batch point" service=kapacitor task_master=main task=kapacitor-task-roll-avg-vpc1-az1a-msql-track01-vm-sys-disk-used node=log2 prefix= name=disk group= field_used=16671768576 time=2018-03-11T23:05:20Z
ts=2018-03-12T00:05:21.344+01:00 lvl=info msg="end batch" service=kapacitor task_master=main task=kapacitor-task-roll-avg-vpc1-az1a-msql-track01-vm-sys-disk-used node=log2 prefix= name=disk group= time=2018-03-12T00:05:21.327512952+01:00
ts=2018-03-12T00:05:21.345+01:00 lvl=info msg=point service=kapacitor task_master=main task=kapacitor-task-roll-avg-vpc1-az1a-msql-track01-vm-sys-disk-used node=log4 prefix= name=disk db= rp= group= field_used=16671765845.333334 time=2018-03-12T00:05:21.327512952+01:00
```
## kapacitor show task output (directly after above logging output) ##
```ID: kapacitor-task-roll-avg-vpc1-az1a-msql-track01-vm-sys-disk-used
Error:
Template:
Type: batch
Status: enabled
Executing: true
Created: 10 Mar 18 13:55 CET
Modified: 12 Mar 18 00:04 CET
LastEnabled: 12 Mar 18 00:04 CET
Databases Retention Policies: ["vpcx_central_monitoring"."rp_weeks_12"]
TICKscript:
batch
|query('''SELECT "used" FROM "vpcx_central_monitoring"."rp_weeks_12".disk WHERE ("host" = 'vpc1-az1a-msql-track01')''')
.period(1m)
.every(1m)
.fill(0.0)
|log()
|mean('used')
.as('used')
|log()
|alert()
.warn(lambda: "used" > 1.0)
.log('/var/log/kapacitor/vm-sys.log')
.message('{{ .Level }} for {{ .ID }}')
|log()
DOT:
digraph kapacitor-task-roll-avg-vpc1-az1a-msql-track01-vm-sys-disk-used {
graph [throughput="0.00 batches/s"];
query1 [avg_exec_time_ns="0s" batches_queried="1" errors="0" points_queried="6" working_cardinality="0" ];
query1 -> log2 [processed="1"];
log2 [avg_exec_time_ns="0s" errors="0" working_cardinality="0" ];
log2 -> mean3 [processed="1"];
mean3 [avg_exec_time_ns="0s" errors="0" working_cardinality="1" ];
mean3 -> log4 [processed="1"];
log4 [avg_exec_time_ns="632.45µs" errors="0" working_cardinality="0" ];
log4 -> alert5 [processed="1"];
alert5 [alerts_triggered="0" avg_exec_time_ns="0s" crits_triggered="0" errors="0" infos_triggered="0" oks_triggered="0" warns_triggered="0" working_cardinality="1" ];
alert5 -> log6 [processed="0"];
log6 [avg_exec_time_ns="0s" errors="0" working_cardinality="0" ];
}
```
Any help to point me in the right direction would be greatly appreciated!
Contributor guide
Research direction
Start with the supplied batch TICKscript, Kapacitor log output, and `kapacitor show task` output, comparing the query, mean, alert, and final log nodes. Reproduce the intermittent behavior with the listed Kapacitor, InfluxDB, and Telegraf versions; done means the alert executes consistently or the failure condition is identified and covered by a regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100