influxdata / influxdata/kapacitor

Stream node seems to get some data not properly.

Open
#2,334 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

Hi,

One of my alerts requires Kapacitor to count the number of host on my platform every 10s. With a batch node I have done it with the following tick script:

```
dbrp "telegraf"."autogen"

var nb_node_threshold = 0

var nb_node = batch
|query('''
SELECT count("usage_idle")
FROM "telegraf"."autogen"."cpu"
WHERE "cpu" = 'cpu-total'
''')
.period(10s)
.every(10s)
.fill(0)

var alert = nb_node
|alert()
.crit(lambda: "count" > nb_node_threshold)
.message('Nb of nodes {{ index .Fields "count" }}')
.log('/var/log/kapacitor/elasticitynode.log')
```

At first, for some reasons, I wanted to do it with a stream node. I have made the following tick script to do it:

```
dbrp "telegraf"."autogen"

// Parameters
var nb_node_threshold = 0
var period = 10s
var every = 10s

var nb_nodes = stream
|from()
.measurement('cpu')
.where(lambda: "cpu" == 'cpu-total')
|window()
.align()
.period(period)
.every(every)
|count('usage_user')
.as('value')

var alert = nb_nodes
|alert()
.crit(lambda: "value" > nb_node_threshold)
.message('Nb of nodes {{ index .Fields "value" }}')
.log('/var/log/kapacitor/elasticity.log')
```
Reading Kapacitor documentation, I thought this "stream" alert was equivalent to the "batch" alert above. But most of the time, this alert doesn't count the right number of host.

By digging a bit into that issue, I found that sometimes the stream node gets, for a single time, two values for a given field. To make things clear here are two json extract from the log output of Kapacitor:

```
{ "id":"cpu:host=XXX",
"message":"Nb of nodes 99.64929863291364 Time : 2020-05-04 07:54:11.501936989 +0000 UTC",
"details":"{\u0026#34;Name\u0026#34;:\u0026#34;cpu\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;test_alert\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;host=XXX\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;host\u0026#34;:\u0026#34;XXX\u0026#34;},\u0026#34;ServerInfo\u0026#34;:{\u0026#34;Hostname\u0026#34;:\u0026#34;kapacitor-rs-7f4db45749-9vphp\u0026#34;,\u0026#34;ClusterID\u0026#34;:\u0026#34;fd3a9be1-0f81-4c67-a1a0-7c03ff5db5ef\u0026#34;,\u0026#34;ServerID\u0026#34;:\u0026#34;ff6029e9-2f01-4046-aaa6-c9f8a0876cc2\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;cpu:host=XXX\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;usage_guest\u0026#34;:0,\u0026#34;usage_guest_nice\u0026#34;:0,\u0026#34;usage_idle\u0026#34;:99.64929863291364,\u0026#34;usage_iowait\u0026#34;:0,\u0026#34;usage_irq\u0026#34;:0,\u0026#34;usage_nice\u0026#34;:0,\u0026#34;usage_softirq\u0026#34;:0,\u0026#34;usage_steal\u0026#34;:0,\u0026#34;usage_system\u0026#34;:0.1503006012459435,\u0026#34;usage_user\u0026#34;:0.20040080170986158},\u0026#34;Level\u0026#34;:\u0026#34;CRITICAL\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2020-05-04T07:54:11.501936989Z\u0026#34;,\u0026#34;Duration\u0026#34;:10000000000,\u0026#34;Message\u0026#34;:\u0026#34;Nb of nodes 99.64929863291364 Time : 2020-05-04 07:54:11.501936989 \u0026#43;0000 UTC\u0026#34;}\n",
"time":"2020-05-04T07:54:11.501936989Z",
"duration":10000000000,
"level":"CRITICAL",
"data":{ "series":[ { "name":"cpu",
"tags":{
"host":"XXX"

},
"columns":[
"time",
"cpu",
[...]
],
"values":[ [
"2020-05-04T07:54:11.501936989Z",
"cpu-total",
[...]

]
]
}
]
},
"previousLevel":"CRITICAL",
"recoverable":true
}
```
That json seems to be correct, the time above duration is identical to the time just below "values"
But in the following json, they are two values of each field for a single time:
```
{ "id":"cpu:host=XXX",
"message":"Nb of nodes 99.7996995537891 Time : 2020-05-04 08:58:42 +0000 UTC",
"details":"{\u0026#34;Name\u0026#34;:\u0026#34;cpu\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;test_alert\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;host=XXX\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;host\u0026#34;:\u0026#34;XXX\u0026#34;},\u0026#34;ServerInfo\u0026#34;:{\u0026#34;Hostname\u0026#34;:\u0026#34;kapacitor-rs-7f4db45749-9vphp\u0026#34;,\u0026#34;ClusterID\u0026#34;:\u0026#34;fd3a9be1-0f81-4c67-a1a0-7c03ff5db5ef\u0026#34;,\u0026#34;ServerID\u0026#34;:\u0026#34;ff6029e9-2f01-4046-aaa6-c9f8a0876cc2\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;cpu:host=XXX\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;usage_guest\u0026#34;:0,\u0026#34;usage_guest_nice\u0026#34;:0,\u0026#34;usage_idle\u0026#34;:99.7996995537891,\u0026#34;usage_iowait\u0026#34;:0,\u0026#34;usage_irq\u0026#34;:0,\u0026#34;usage_nice\u0026#34;:0,\u0026#34;usage_softirq\u0026#34;:0,\u0026#34;usage_steal\u0026#34;:0,\u0026#34;usage_system\u0026#34;:0.10015022536438868,\u0026#34;usage_user\u0026#34;:0.10015022532795424},\u0026#34;Level\u0026#34;:\u0026#34;CRITICAL\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2020-05-04T08:58:42Z\u0026#34;,\u0026#34;Duration\u0026#34;:20000000000,\u0026#34;Message\u0026#34;:\u0026#34;Nb of nodes 99.7996995537891 Time : 2020-05-04 08:58:42 \u0026#43;0000 UTC\u0026#34;}\n",
"time":"2020-05-04T08:58:42Z",
"duration":20000000000,
"level":"CRITICAL",
"data":{ "series":[ { "name":"cpu",
"tags":{
"host":"XXX"

},
"columns":[
"time",
"cpu",
[...]

],
"values":[ [
"2020-05-04T08:58:42Z",
"cpu-total",
[...]

],
[
"2020-05-04T08:58:52Z",
"cpu-total",
[...]

]
]
}
]
},
"previousLevel":"CRITICAL",
"recoverable":true
}
```

Where does that issue comes from ? Is that my tick script that is incorrect ?

Thanks

Contributor guide

Open the contributing guide

Research direction

Start with the stream tick script's from(), window(), and count() pipeline, then compare its 10-second windows with the batch query and the two alert log extracts. Reproduce the duplicate values and determine whether the behavior comes from the script or stream processing; done means the cause and any required correction are documented or tested.

Written by the indexing model from the issue text.

Assessment

Domain
observability-sre, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.