influxdata / influxdata/kapacitor

Deadman OK Reports Incorrect Number of Points Emitted

Open
#1,969 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

OS: Linux version 4.1.10-202.fc22.x86_64 (Red Hat 5.3.1-2)
Kapacitor 1.2.0 (git: master 5408057e5a3493d3b5bd38d5d535ea45b587f8ff)

I have a deadman set up on a 10 minute timer. If the number of points seen falls below 100 then we trigger. The TICKscript looks like this:
```
TICKscript:
var s = stream
|from()
.database('sysQueries')
.retentionPolicy('autogen')
.measurement('fireMeasure')
.where(lambda: "hostBox" =~ /^(fireHost1|fireHost2)$/)
.groupBy('hostBox')

s
|deadman(100.0, 10m)
.id('fireAlert {{ .Group }}')
.message('{{ .ID }} is {{ .Level }}: {{ index .Fields "emitted" | printf "%.0f" }} points seen in last 10m')
.stateChangesOnly()
.log('/tmp/fireAlert.log')
.mode(0644)
```

We correctly triggered CRITICAL in a 10m interval and reset appropriately to OK in following 10m interval. However the numbers of points reported in the OK interval is inconsistent with the number we are actually seeing and recording in our influx db.
From fireAlert.log:
```
{"id":"fireAlert hostBox=fireHost1","message":"fireAlert hostBox=fireHost1 is CRITICAL: 76 points seen in last 10m","details":"{\u0026#34;Name\u0026#34;:\u0026#34;stats\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;fireAlert\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;hostBox=fireHost1\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;hostBox\u0026#34;:\u0026#34;fireHost1\u0026#34;,\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;fireAlert hostBox=fireHost1\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;emitted\u0026#34;:76},\u0026#34;Level\u0026#34;:\u0026#34;CRITICAL\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2018-06-18T14:20:00Z\u0026#34;,\u0026#34;Message\u0026#34;:\u0026#34;fireAlert hostBox=fireHost1 is CRITICAL: 76 points seen in last 10m\u0026#34;}\n","time":"2018-06-18T14:20:00Z","duration":0,"level":"CRITICAL","data":{"Series":[{"name":"stats","tags":{"hostBox":"fireHost1"},"columns":["time","emitted"],"values":[["2018-06-18T14:20:00Z",76]]}],"Messages":null,"Err":null}}
{"id":"fireAlert hostBox=fireHost1","message":"fireAlert hostBox=fireHost1 is OK: 4622 points seen in last 10m","details":"{\u0026#34;Name\u0026#34;:\u0026#34;stats\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;fireAlert\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;hostBox=fireHost1\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;hostBox\u0026#34;:\u0026#34;fireHost1\u0026#34;,\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;fireAlert hostBox=fireHost1\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;emitted\u0026#34;:4622},\u0026#34;Level\u0026#34;:\u0026#34;OK\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2018-06-18T14:30:00Z\u0026#34;,\u0026#34;Message\u0026#34;:\u0026#34;fireAlert hostBox=fireHost1 is OK: 4622 points seen in last 10m\u0026#34;}\n","time":"2018-06-18T14:30:00Z","duration":600000000000,"level":"OK","data":{"Series":[{"name":"stats","tags":{"hostBox":"fireHost1"},"columns":["time","emitted"],"values":[["2018-06-18T14:30:00Z",4622]]}],"Messages":null,"Err":null}}
```

The influx DB that has this data shows the correct numbers.
```
> select count(num) from fireMeasure where host =~ /^fireHost1$/ and time >= '2018-06-18T14:20:00Z' and time < '2018-06-18T14:40:00Z' group by time(10m)
name: fireMeasure
time count
---- -----
2018-06-18T14:20:00Z 76
2018-06-18T14:30:00Z 191
```

As you can see, the fireAlert.log shows CRITICAL and correctly reports 76 but OK incorrectly reports 4622.

Contributor guide

Open the contributing guide

Research direction

Start with the deadman operator and reproduce the supplied TICKscript and interval counts, comparing the alert's emitted value with the InfluxDB results. Done means the OK alert reports the number of points emitted during its own 10-minute interval rather than the inflated value shown in the log.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability-sre, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.