influxdata / influxdata/kapacitor
Kapacitor "show <task name>" shows no data points consumed, but "stats ingress" and strace do
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
**OS:** CentOS Linux release 7.2.1511 (Core)
**Kapacitor and InfluxDB Versions:** 1.0.0
Built and installed from your source (no special/weird make options specified):
https://dl.influxdata.com/kapacitor/releases/kapacitor-1.0.0_linux_amd64.tar.gz
## Issue:
`kapacitor list tasks` shows the task running and enabled:
```
ID Type Status Executing Databases and Retention Policies
core-iperf3-bandwidth batch enabled true ["pipe-apps"."pipe-apps"]
```
`kapacitor stats ingress` shows points received:
```
pipe-apps pipe-apps iperf3 145
```
`kapacitor show core-iperf3-bandwidth` shows 0 batches collected:
```
ID: core-iperf3-bandwidth
Error:
Template:
Type: batch
Status: enabled
Executing: true
Created: 23 Jun 17 17:59 UTC
Modified: 23 Jun 17 19:16 UTC
LastEnabled: 23 Jun 17 19:16 UTC
Databases Retention Policies: ["pipe-apps"."pipe-apps"]
TICKscript:
// core-iper3-bandwidth
// Email parameters
var emailAddress =
var msg_details = '''
Kapacitor
Notification Type:
{{ if eq .Level "INFO" }}Info{{ else }}Alert{{ end }}
iPerf3 Client Instance:
{{ index .Tags "host" }}
AWS Region:
{{ index .Tags "region" }}
Kapacitor Task Name:
{{ .TaskName }}
Check State:
{{ .Level }}
Event Time:
{{ .Time }}
'''
// Parameters
var critBits = 104857600
var window = 60m
var frequency = 15m
var buckets = 30m
var renotify = 15m
// Dataframe
var data = batch
|query('''
SELECT mean(sent_bps)
AS stat
FROM "pipe-apps"."pipe-apps"."iperf3"
WHERE "host" =~ /.*/
''')
.period(window)
.every(frequency)
.groupBy(time(buckets), 'host')
// Thresholds
var alert = data
|alert()
.id('{{ index .Tags "host" }}')
.message('Average network bandwidth for {{ .ID }} is {{ .Level }}: {{ index .Fields "stat" }}/Bits per sec')
.details(msg_details)
.stateChangesOnly(renotify)
.crit(lambda: "stat" < critBits)
// Alert
alert
.email(emailAddress)
DOT:
digraph core-iperf3-bandwidth {
graph [throughput="0.00 batches/s"];
query1 [avg_exec_time_ns="0" batches_queried="9" connect_errors="0" points_queried="26" query_errors="0" ];
query1 -> alert2 [processed="9"];
alert2 [alerts_triggered="0" avg_exec_time_ns="0" crits_triggered="0" infos_triggered="0" oks_triggered="0" warns_triggered="0" ];
}
```
Also, when I strace and filter on `read()` calls, I see the kapacitor process is reading the query results in question from our influxdb instance:
```
[pid 26143] <... read resumed> "POST /write?consistency=&db=&precision=ns&rp= HTTP/1.1\r\nHost: \r\nUser-Agent: InfluxDBClient\r\nContent-Length: 966\r\nContent-Type: \r\nAccept-Encoding: gzip\r\n\r\niperf3,dirty_data=false,host=,instance_id=,receiving_server=,region=us-east-1 received_bps=1039223000,sent_bps=1040483000 1498250287000000000\niperf3,dirty_data=false,host=,instance_id=,receiving_server=,region=us-east-1 received_bps=1055083000,sent_bps=1056186000 1498250287000000000\niperf3,dirty_data=false,host=,instance_id=,receiving_server=,region=us-east-1 received_bps=60238092,sent_bps=61656726 1498250287000000000\niperf3,dirty_data=true,host=,instance_id=,receiving_server=,region=us-east-1 received_bps=0,sent_bps=78082 1498250287000000000\niperf3,dirty_data=true,host=,instance_id=,receiving_server=,region=us-east-1 received_bps=0,sent_bps=78083 1498250287000000000\n", 4096) = 1192
```
Contributor guide
Research direction
Start by reproducing the discrepancy with `kapacitor stats ingress`, `kapacitor show core-iperf3-bandwidth`, and the provided batch query, then compare the DOT statistics with the InfluxDB response observed through `strace`. Done means identifying why ingested points and query results are not reflected as consumed batches and documenting or testing the corrected behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100