influxdata / influxdata/kapacitor

Alert Node failing intermittently using Batch script

Open
#1,849 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

I am using a Batch script to try and create a simple alert based on disk utilization. The below script functions part of the time and I have yet to discover exactly why it would fail intermittently. Between attempts I define/disable/enable the script. Clearly the data is present inside InfluxDB, and the Mean operation is run succesfully.

- Telegraf is used as Agent to send data into InfluxDB
- Kapacitor is configured globally using `state-changes-only = false`
- Kapacitor v1.4.0
- InfluxDB v1.4.3
- Telegraf v1.5.2-1

## Remote host Telegraf configuration ##
```[[inputs.disk]]
## By default, telegraf gather stats for all mountpoints.
## Setting mountpoints will restrict the stats to the specified mountpoints.
mount_points = ["/"]
```

## Script Configuration ##
```
batch
|query('''SELECT "used" FROM "vpcx_central_monitoring"."rp_weeks_12".disk WHERE ("host" = 'vpc1-az1a-msql-track01')''')
.period(1m)
.every(1m)
.fill(0.0)

|log()

|mean('used')
.as('used')

|log()

|alert()
.warn(lambda: "used" > 1.0)
.log('/var/log/kapacitor/vm-sys.log')
.message('{{ .Level }} for {{ .ID }}')

|log()
```

## Output from kapacitor.log ##
```ts=2018-03-12T00:05:21.341+01:00 lvl=info msg="begin batch" service=kapacitor task_master=main task=kapacitor-task-roll-avg-vpc1-az1a-msql-track01-vm-sys-disk-used node=log2 prefix= name=disk group= time=2018-03-12T00:05:21.327512952+01:00
ts=2018-03-12T00:05:21.342+01:00 lvl=info msg="batch point" service=kapacitor task_master=main task=kapacitor-task-roll-avg-vpc1-az1a-msql-track01-vm-sys-disk-used node=log2 prefix= name=disk group= field_used=16671760384 time=2018-03-11T23:04:30Z
ts=2018-03-12T00:05:21.342+01:00 lvl=info msg="batch point" service=kapacitor task_master=main task=kapacitor-task-roll-avg-vpc1-az1a-msql-track01-vm-sys-disk-used node=log2 prefix= name=disk group= field_used=16671764480 time=2018-03-11T23:04:40Z
ts=2018-03-12T00:05:21.342+01:00 lvl=info msg="batch point" service=kapacitor task_master=main task=kapacitor-task-roll-avg-vpc1-az1a-msql-track01-vm-sys-disk-used node=log2 prefix= name=disk group= field_used=16671764480 time=2018-03-11T23:04:50Z
ts=2018-03-12T00:05:21.343+01:00 lvl=info msg="batch point" service=kapacitor task_master=main task=kapacitor-task-roll-avg-vpc1-az1a-msql-track01-vm-sys-disk-used node=log2 prefix= name=disk group= field_used=16671768576 time=2018-03-11T23:05:00Z
ts=2018-03-12T00:05:21.343+01:00 lvl=info msg="batch point" service=kapacitor task_master=main task=kapacitor-task-roll-avg-vpc1-az1a-msql-track01-vm-sys-disk-used node=log2 prefix= name=disk group= field_used=16671768576 time=2018-03-11T23:05:10Z
ts=2018-03-12T00:05:21.344+01:00 lvl=info msg="batch point" service=kapacitor task_master=main task=kapacitor-task-roll-avg-vpc1-az1a-msql-track01-vm-sys-disk-used node=log2 prefix= name=disk group= field_used=16671768576 time=2018-03-11T23:05:20Z
ts=2018-03-12T00:05:21.344+01:00 lvl=info msg="end batch" service=kapacitor task_master=main task=kapacitor-task-roll-avg-vpc1-az1a-msql-track01-vm-sys-disk-used node=log2 prefix= name=disk group= time=2018-03-12T00:05:21.327512952+01:00
ts=2018-03-12T00:05:21.345+01:00 lvl=info msg=point service=kapacitor task_master=main task=kapacitor-task-roll-avg-vpc1-az1a-msql-track01-vm-sys-disk-used node=log4 prefix= name=disk db= rp= group= field_used=16671765845.333334 time=2018-03-12T00:05:21.327512952+01:00
```

## kapacitor show task output (directly after above logging output) ##
```ID: kapacitor-task-roll-avg-vpc1-az1a-msql-track01-vm-sys-disk-used
Error:
Template:
Type: batch
Status: enabled
Executing: true
Created: 10 Mar 18 13:55 CET
Modified: 12 Mar 18 00:04 CET
LastEnabled: 12 Mar 18 00:04 CET
Databases Retention Policies: ["vpcx_central_monitoring"."rp_weeks_12"]
TICKscript:
batch
|query('''SELECT "used" FROM "vpcx_central_monitoring"."rp_weeks_12".disk WHERE ("host" = 'vpc1-az1a-msql-track01')''')
.period(1m)
.every(1m)
.fill(0.0)
|log()
|mean('used')
.as('used')
|log()
|alert()
.warn(lambda: "used" > 1.0)
.log('/var/log/kapacitor/vm-sys.log')
.message('{{ .Level }} for {{ .ID }}')
|log()

DOT:
digraph kapacitor-task-roll-avg-vpc1-az1a-msql-track01-vm-sys-disk-used {
graph [throughput="0.00 batches/s"];

query1 [avg_exec_time_ns="0s" batches_queried="1" errors="0" points_queried="6" working_cardinality="0" ];
query1 -> log2 [processed="1"];

log2 [avg_exec_time_ns="0s" errors="0" working_cardinality="0" ];
log2 -> mean3 [processed="1"];

mean3 [avg_exec_time_ns="0s" errors="0" working_cardinality="1" ];
mean3 -> log4 [processed="1"];

log4 [avg_exec_time_ns="632.45µs" errors="0" working_cardinality="0" ];
log4 -> alert5 [processed="1"];

alert5 [alerts_triggered="0" avg_exec_time_ns="0s" crits_triggered="0" errors="0" infos_triggered="0" oks_triggered="0" warns_triggered="0" working_cardinality="1" ];
alert5 -> log6 [processed="0"];

log6 [avg_exec_time_ns="0s" errors="0" working_cardinality="0" ];
}
```

Any help to point me in the right direction would be greatly appreciated!

Contributor guide

Open the contributing guide

Research direction

Start with the supplied batch TICKscript, Kapacitor log output, and `kapacitor show task` output, comparing the query, mean, alert, and final log nodes. Reproduce the intermittent behavior with the listed Kapacitor, InfluxDB, and Telegraf versions; done means the alert executes consistently or the failure condition is identified and covered by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.