influxdata / influxdata/kapacitor
Kapacitor window missing events and emitting at irregular intervals
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
We have been trying to use kapacitor to perform aggregations. The data we push to kapacitor can be unordered, i.e time of events can be unordered. So to check for any data loss we performed a small experiment with below tick script
```
stream
|from()
.database('db_name')
.measurement('fs')
.groupBy('host','group','region')
|window()
.period(5m)
.every(5m)
|log()
.level('DEBUG')
|count('status')
.as('count_2xx')
|influxDBOut()
.database('db_name')
.retentionPolicy('rp')
.measurement('count_status')
.precision('s')
```
And observed the following in kapacitor log:
```
ts=2020-02-03T12:38:13.477Z lvl=debug msg="begin batch" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= tag_region= tag_group= tag_host= time=2020-01-18T07:47:38Z
ts=2020-02-03T12:38:13.477Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-18T07:42:38Z
ts=2020-02-03T12:38:13.477Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_status=200 field_db_runtime=1.76 time=2020-01-18T07:41:38Z
ts=2020-02-03T12:38:13.477Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-18T07:41:38Z
ts=2020-02-03T12:38:13.477Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-18T07:41:38Z
ts=2020-02-03T12:38:13.477Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-18T07:41:38Z
ts=2020-02-03T12:38:13.477Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-18T07:41:38Z
ts=2020-02-03T12:38:13.477Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-18T07:41:38Z
ts=2020-02-03T12:38:13.477Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-18T07:41:38Z
ts=2020-02-03T12:38:13.477Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-18T07:41:38Z
ts=2020-02-03T12:38:13.477Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_status=200 field_db_runtime=1.76 time=2020-01-18T07:41:38Z
ts=2020-02-03T12:38:13.477Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-18T07:41:38Z
ts=2020-02-03T12:38:13.477Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-18T07:41:38Z
ts=2020-02-03T12:38:13.477Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-18T07:41:38Z
ts=2020-02-03T12:38:13.477Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-18T07:41:38Z
ts=2020-02-03T12:38:13.477Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-18T07:41:38Z
ts=2020-02-03T12:38:13.477Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-18T07:41:38Z
ts=2020-02-03T12:38:13.477Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-18T07:41:38Z
ts=2020-02-03T12:38:13.477Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-18T07:41:38Z
ts=2020-02-03T12:38:13.477Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-18T07:41:38Z
ts=2020-02-03T12:38:13.477Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-18T07:41:38Z
ts=2020-02-03T12:38:13.477Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-18T07:41:38Z
ts=2020-02-03T12:38:13.477Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_status=200 field_db_runtime=1.76 time=2020-01-18T07:41:38Z
ts=2020-02-03T12:38:13.477Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-18T07:41:38Z
ts=2020-02-03T12:38:13.477Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-18T07:41:38Z
ts=2020-02-03T12:38:13.477Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-18T07:41:38Z
ts=2020-02-03T12:38:13.477Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-18T07:41:38Z
ts=2020-02-03T12:38:13.477Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-18T07:43:38Z
ts=2020-02-03T12:38:13.477Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-18T07:44:38Z
ts=2020-02-03T12:38:13.477Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_status=200 field_db_runtime=1.76 time=2020-01-18T07:45:38Z
ts=2020-02-03T12:38:13.477Z lvl=debug msg="end batch" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= tag_region= tag_group= tag_host= time=2020-01-18T07:47:38Z
ts=2020-02-03T12:38:19.694Z lvl=debug msg="begin batch" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= tag_region= tag_group= tag_host= time=2020-01-19T07:46:38Z
ts=2020-02-03T12:38:19.694Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=204 time=2020-01-19T07:41:38Z
ts=2020-02-03T12:38:19.694Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.86 field_status=200 time=2020-01-19T07:41:38Z
ts=2020-02-03T12:38:19.694Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.86 field_status=200 time=2020-01-19T07:41:38Z
ts=2020-02-03T12:38:19.694Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-19T07:41:38Z
ts=2020-02-03T12:38:19.694Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_status=200 field_db_runtime=1.76 time=2020-01-19T07:41:38Z
ts=2020-02-03T12:38:19.694Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.86 field_status=200 time=2020-01-19T07:41:38Z
ts=2020-02-03T12:38:19.694Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_status=200 field_db_runtime=1.76 time=2020-01-19T07:41:38Z
ts=2020-02-03T12:38:19.695Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.86 field_status=200 time=2020-01-19T07:41:38Z
ts=2020-02-03T12:38:19.695Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-19T07:41:38Z
ts=2020-02-03T12:38:19.695Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.86 field_status=200 time=2020-01-19T07:41:38Z
ts=2020-02-03T12:38:19.695Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-19T07:41:38Z
ts=2020-02-03T12:38:19.695Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-18T07:41:38Z
ts=2020-02-03T12:38:19.695Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-18T07:41:38Z
ts=2020-02-03T12:38:19.695Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-18T07:41:38Z
ts=2020-02-03T12:38:19.695Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-18T07:41:38Z
ts=2020-02-03T12:38:19.695Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-18T07:41:38Z
ts=2020-02-03T12:38:19.695Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-18T07:41:38Z
ts=2020-02-03T12:38:19.695Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-18T07:41:38Z
ts=2020-02-03T12:38:19.695Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-18T07:41:38Z
ts=2020-02-03T12:38:19.695Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-18T07:41:38Z
ts=2020-02-03T12:38:19.695Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-18T07:41:38Z
ts=2020-02-03T12:38:19.695Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-18T07:41:38Z
ts=2020-02-03T12:38:19.695Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-18T07:41:38Z
ts=2020-02-03T12:38:19.695Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-18T07:41:38Z
ts=2020-02-03T12:38:19.695Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-18T07:41:38Z
ts=2020-02-03T12:38:19.695Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-18T07:41:38Z
ts=2020-02-03T12:38:19.695Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-18T07:41:38Z
ts=2020-02-03T12:38:19.695Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-18T07:41:38Z
ts=2020-02-03T12:38:19.695Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-18T07:41:38Z
ts=2020-02-03T12:38:19.695Z lvl=debug msg="batch point" service=kapacitor task_master=main task=test_alert node=log3 prefix= name=fs group=group=,host=,region= field_db_runtime=1.76 field_status=200 time=2020-01-18T07:41:38Z
```
If we observe in above log line the second `begin batch` had points outside its batch frame. This made us assume we dont miss events even if the order of events is not proper.
But When we put it on production, We saw the count of events mismatch from actual events received at kapacitor to events written to influx.
q1. Does the unordered events cause issue?
q2. We also observed the emit for every 5m is not happening every 5m as well. SO what does emit actually do
It would be really helpful if you could provide us with an answer @nathanielc
Contributor guide
Assessment
This issue has not been assessed yet.