influxdata / influxdata/kapacitor
Stream node seems to get some data not properly.
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
Hi,
One of my alerts requires Kapacitor to count the number of host on my platform every 10s. With a batch node I have done it with the following tick script:
```
dbrp "telegraf"."autogen"
var nb_node_threshold = 0
var nb_node = batch
|query('''
SELECT count("usage_idle")
FROM "telegraf"."autogen"."cpu"
WHERE "cpu" = 'cpu-total'
''')
.period(10s)
.every(10s)
.fill(0)
var alert = nb_node
|alert()
.crit(lambda: "count" > nb_node_threshold)
.message('Nb of nodes {{ index .Fields "count" }}')
.log('/var/log/kapacitor/elasticitynode.log')
```
At first, for some reasons, I wanted to do it with a stream node. I have made the following tick script to do it:
```
dbrp "telegraf"."autogen"
// Parameters
var nb_node_threshold = 0
var period = 10s
var every = 10s
var nb_nodes = stream
|from()
.measurement('cpu')
.where(lambda: "cpu" == 'cpu-total')
|window()
.align()
.period(period)
.every(every)
|count('usage_user')
.as('value')
var alert = nb_nodes
|alert()
.crit(lambda: "value" > nb_node_threshold)
.message('Nb of nodes {{ index .Fields "value" }}')
.log('/var/log/kapacitor/elasticity.log')
```
Reading Kapacitor documentation, I thought this "stream" alert was equivalent to the "batch" alert above. But most of the time, this alert doesn't count the right number of host.
By digging a bit into that issue, I found that sometimes the stream node gets, for a single time, two values for a given field. To make things clear here are two json extract from the log output of Kapacitor:
```
{ "id":"cpu:host=XXX",
"message":"Nb of nodes 99.64929863291364 Time : 2020-05-04 07:54:11.501936989 +0000 UTC",
"details":"{\u0026#34;Name\u0026#34;:\u0026#34;cpu\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;test_alert\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;host=XXX\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;host\u0026#34;:\u0026#34;XXX\u0026#34;},\u0026#34;ServerInfo\u0026#34;:{\u0026#34;Hostname\u0026#34;:\u0026#34;kapacitor-rs-7f4db45749-9vphp\u0026#34;,\u0026#34;ClusterID\u0026#34;:\u0026#34;fd3a9be1-0f81-4c67-a1a0-7c03ff5db5ef\u0026#34;,\u0026#34;ServerID\u0026#34;:\u0026#34;ff6029e9-2f01-4046-aaa6-c9f8a0876cc2\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;cpu:host=XXX\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;usage_guest\u0026#34;:0,\u0026#34;usage_guest_nice\u0026#34;:0,\u0026#34;usage_idle\u0026#34;:99.64929863291364,\u0026#34;usage_iowait\u0026#34;:0,\u0026#34;usage_irq\u0026#34;:0,\u0026#34;usage_nice\u0026#34;:0,\u0026#34;usage_softirq\u0026#34;:0,\u0026#34;usage_steal\u0026#34;:0,\u0026#34;usage_system\u0026#34;:0.1503006012459435,\u0026#34;usage_user\u0026#34;:0.20040080170986158},\u0026#34;Level\u0026#34;:\u0026#34;CRITICAL\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2020-05-04T07:54:11.501936989Z\u0026#34;,\u0026#34;Duration\u0026#34;:10000000000,\u0026#34;Message\u0026#34;:\u0026#34;Nb of nodes 99.64929863291364 Time : 2020-05-04 07:54:11.501936989 \u0026#43;0000 UTC\u0026#34;}\n",
"time":"2020-05-04T07:54:11.501936989Z",
"duration":10000000000,
"level":"CRITICAL",
"data":{ "series":[ { "name":"cpu",
"tags":{
"host":"XXX"
},
"columns":[
"time",
"cpu",
[...]
],
"values":[ [
"2020-05-04T07:54:11.501936989Z",
"cpu-total",
[...]
]
]
}
]
},
"previousLevel":"CRITICAL",
"recoverable":true
}
```
That json seems to be correct, the time above duration is identical to the time just below "values"
But in the following json, they are two values of each field for a single time:
```
{ "id":"cpu:host=XXX",
"message":"Nb of nodes 99.7996995537891 Time : 2020-05-04 08:58:42 +0000 UTC",
"details":"{\u0026#34;Name\u0026#34;:\u0026#34;cpu\u0026#34;,\u0026#34;TaskName\u0026#34;:\u0026#34;test_alert\u0026#34;,\u0026#34;Group\u0026#34;:\u0026#34;host=XXX\u0026#34;,\u0026#34;Tags\u0026#34;:{\u0026#34;host\u0026#34;:\u0026#34;XXX\u0026#34;},\u0026#34;ServerInfo\u0026#34;:{\u0026#34;Hostname\u0026#34;:\u0026#34;kapacitor-rs-7f4db45749-9vphp\u0026#34;,\u0026#34;ClusterID\u0026#34;:\u0026#34;fd3a9be1-0f81-4c67-a1a0-7c03ff5db5ef\u0026#34;,\u0026#34;ServerID\u0026#34;:\u0026#34;ff6029e9-2f01-4046-aaa6-c9f8a0876cc2\u0026#34;},\u0026#34;ID\u0026#34;:\u0026#34;cpu:host=XXX\u0026#34;,\u0026#34;Fields\u0026#34;:{\u0026#34;usage_guest\u0026#34;:0,\u0026#34;usage_guest_nice\u0026#34;:0,\u0026#34;usage_idle\u0026#34;:99.7996995537891,\u0026#34;usage_iowait\u0026#34;:0,\u0026#34;usage_irq\u0026#34;:0,\u0026#34;usage_nice\u0026#34;:0,\u0026#34;usage_softirq\u0026#34;:0,\u0026#34;usage_steal\u0026#34;:0,\u0026#34;usage_system\u0026#34;:0.10015022536438868,\u0026#34;usage_user\u0026#34;:0.10015022532795424},\u0026#34;Level\u0026#34;:\u0026#34;CRITICAL\u0026#34;,\u0026#34;Time\u0026#34;:\u0026#34;2020-05-04T08:58:42Z\u0026#34;,\u0026#34;Duration\u0026#34;:20000000000,\u0026#34;Message\u0026#34;:\u0026#34;Nb of nodes 99.7996995537891 Time : 2020-05-04 08:58:42 \u0026#43;0000 UTC\u0026#34;}\n",
"time":"2020-05-04T08:58:42Z",
"duration":20000000000,
"level":"CRITICAL",
"data":{ "series":[ { "name":"cpu",
"tags":{
"host":"XXX"
},
"columns":[
"time",
"cpu",
[...]
],
"values":[ [
"2020-05-04T08:58:42Z",
"cpu-total",
[...]
],
[
"2020-05-04T08:58:52Z",
"cpu-total",
[...]
]
]
}
]
},
"previousLevel":"CRITICAL",
"recoverable":true
}
```
Where does that issue comes from ? Is that my tick script that is incorrect ?
Thanks
Contributor guide
Research direction
Start with the stream tick script's from(), window(), and count() pipeline, then compare its 10-second windows with the batch query and the two alert log extracts. Reproduce the duplicate values and determine whether the behavior comes from the script or stream processing; done means the cause and any required correction are documented or tested.
Written by the indexing model from the issue text.
Assessment
- Domain
- observability-sre, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100