influxdata / influxdata/kapacitor
union node not passing points
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
Script:
```
var system = stream|from().measurement('system')|log()
var ntpq = stream|from().measurement('ntpq')|log()
system|union(ntpq)|log()
```
Output:
```
[ntpq_orphan:log4] 2017/02/24 12:27:44 I! {"Name":"ntpq","Database":"telegraf","RetentionPolicy":"autogen","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{"env":"local","group":"uxdb","host":"influxdb","location":"infl","os":"Linux","refid":"10.1.103.20","remote":"10.0.2.12","stratum":"2"},"Fields":{"delay":3.464,"jitter":10.218,"offset":3.573275e+06,"poll":64,"reach":377,"when":38},"Time":"2017-02-24T12:27:38.782Z"}
[ntpq_orphan:log4] 2017/02/24 12:27:44 I! {"Name":"ntpq","Database":"telegraf","RetentionPolicy":"autogen","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{"env":"local","group":"uxdb","host":"influxdb","location":"infl","os":"Linux","refid":"10.1.103.20","remote":"10.0.2.12","stratum":"2"},"Fields":{"delay":3.464,"jitter":10.218,"offset":3.573275e+06,"poll":64,"reach":377,"when":43},"Time":"2017-02-24T12:27:43.817Z"}
[ntpq_orphan:log2] 2017/02/24 12:27:45 I! {"Name":"system","Database":"telegraf","RetentionPolicy":"autogen","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{"env":"local","host":"db1","location":"db1","os":"Linux"},"Fields":{"load1":0,"load15":0.05,"load5":0.01,"n_cpus":3,"n_users":0},"Time":"2017-02-24T14:01:50.131Z"}
[ntpq_orphan:log2] 2017/02/24 12:27:45 I! {"Name":"system","Database":"telegraf","RetentionPolicy":"autogen","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{"env":"local","host":"db1","location":"db1","os":"Linux"},"Fields":{"uptime":230904},"Time":"2017-02-24T14:01:50.131Z"}
[ntpq_orphan:log7] 2017/02/24 12:27:45 I! {"Name":"ntpq","Database":"telegraf","RetentionPolicy":"autogen","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{"env":"local","group":"uxdb","host":"influxdb","location":"infl","os":"Linux","refid":"10.1.103.20","remote":"10.0.2.12","stratum":"2"},"Fields":{"delay":3.464,"jitter":10.218,"offset":3.573275e+06,"poll":64,"reach":377,"when":28},"Time":"2017-02-24T12:27:28.802Z"}
[ntpq_orphan:log7] 2017/02/24 12:27:45 I! {"Name":"ntpq","Database":"telegraf","RetentionPolicy":"autogen","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{"env":"local","group":"uxdb","host":"influxdb","location":"infl","os":"Linux","refid":"10.1.103.20","remote":"10.0.2.12","stratum":"2"},"Fields":{"delay":3.464,"jitter":10.218,"offset":3.573275e+06,"poll":64,"reach":377,"when":33},"Time":"2017-02-24T12:27:33.782Z"}
[ntpq_orphan:log7] 2017/02/24 12:27:45 I! {"Name":"ntpq","Database":"telegraf","RetentionPolicy":"autogen","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{"env":"local","group":"uxdb","host":"influxdb","location":"infl","os":"Linux","refid":"10.1.103.20","remote":"10.0.2.12","stratum":"2"},"Fields":{"delay":3.464,"jitter":10.218,"offset":3.573275e+06,"poll":64,"reach":377,"when":38},"Time":"2017-02-24T12:27:38.782Z"}
[ntpq_orphan:log7] 2017/02/24 12:27:45 I! {"Name":"ntpq","Database":"telegraf","RetentionPolicy":"autogen","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{"env":"local","group":"uxdb","host":"influxdb","location":"infl","os":"Linux","refid":"10.1.103.20","remote":"10.0.2.12","stratum":"2"},"Fields":{"delay":3.464,"jitter":10.218,"offset":3.573275e+06,"poll":64,"reach":377,"when":43},"Time":"2017-02-24T12:27:43.817Z"}
[ntpq_orphan:log2] 2017/02/24 12:27:51 I! {"Name":"system","Database":"telegraf","RetentionPolicy":"autogen","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{"env":"local","host":"jump","location":"jump","os":"Linux"},"Fields":{"load1":0,"load15":0.05,"load5":0.01,"n_cpus":3,"n_users":0},"Time":"2017-02-23T13:18:58.856Z"}
[ntpq_orphan:log2] 2017/02/24 12:27:51 I! {"Name":"system","Database":"telegraf","RetentionPolicy":"autogen","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{"env":"local","host":"jump","location":"jump","os":"Linux"},"Fields":{"uptime":1622654},"Time":"2017-02-23T13:18:58.856Z"}
[ntpq_orphan:log2] 2017/02/24 12:27:54 I! {"Name":"system","Database":"telegraf","RetentionPolicy":"autogen","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{"env":"local","group":"uxdb","host":"influxdb","location":"infl","os":"Linux"},"Fields":{"load1":0.01,"load15":0.05,"load5":0.03,"n_cpus":3,"n_users":4},"Time":"2017-02-24T12:27:44.135Z"}
[ntpq_orphan:log2] 2017/02/24 12:27:54 I! {"Name":"system","Database":"telegraf","RetentionPolicy":"autogen","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{"env":"local","group":"uxdb","host":"influxdb","location":"infl","os":"Linux"},"Fields":{"uptime":1825019},"Time":"2017-02-24T12:27:44.135Z"}
[ntpq_orphan:log4] 2017/02/24 12:27:54 I! {"Name":"ntpq","Database":"telegraf","RetentionPolicy":"autogen","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{"env":"local","group":"uxdb","host":"influxdb","location":"infl","os":"Linux","refid":"10.1.103.20","remote":"10.0.2.12","stratum":"2"},"Fields":{"delay":3.464,"jitter":10.218,"offset":3.573275e+06,"poll":64,"reach":377,"when":48},"Time":"2017-02-24T12:27:48.784Z"}
[ntpq_orphan:log4] 2017/02/24 12:27:54 I! {"Name":"ntpq","Database":"telegraf","RetentionPolicy":"autogen","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{"env":"local","group":"uxdb","host":"influxdb","location":"infl","os":"Linux","refid":"10.1.103.20","remote":"10.0.2.12","stratum":"2"},"Fields":{"delay":3.464,"jitter":10.218,"offset":3.573275e+06,"poll":64,"reach":377,"when":53},"Time":"2017-02-24T12:27:53.781Z"}
[ntpq_orphan:log7] 2017/02/24 12:27:54 I! {"Name":"ntpq","Database":"telegraf","RetentionPolicy":"autogen","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{"env":"local","group":"uxdb","host":"influxdb","location":"infl","os":"Linux","refid":"10.1.103.20","remote":"10.0.2.12","stratum":"2"},"Fields":{"delay":3.464,"jitter":10.218,"offset":3.573275e+06,"poll":64,"reach":377,"when":48},"Time":"2017-02-24T12:27:48.784Z"}
[ntpq_orphan:log7] 2017/02/24 12:27:54 I! {"Name":"ntpq","Database":"telegraf","RetentionPolicy":"autogen","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{"env":"local","group":"uxdb","host":"influxdb","location":"infl","os":"Linux","refid":"10.1.103.20","remote":"10.0.2.12","stratum":"2"},"Fields":{"delay":3.464,"jitter":10.218,"offset":3.573275e+06,"poll":64,"reach":377,"when":53},"Time":"2017-02-24T12:27:53.781Z"}
```
Notice that the only points passing to `log7` are the "ntpq" points. None of the "system" points are making it through.
However if I stop the task, all of a sudden all the "system" points are released.
This looks to be somehow related to the time. I found this issue in my development environment (bunch of vagrant VMs), and the clocks are horribly out of sync. Once I synced all the clocks it started behaving more normal. However the documentation on the union node gives no indication that it cares about time at all. And while my clocks were horribly out of sync, how out of sync do they have to be to break the union node? Will a few milliseconds break it?
This is kapacitor built from current master (d57e7ba).
Contributor guide
Research direction
Start with the reported `system|union(ntpq)|log()` reproduction and compare behavior with synchronized and out-of-sync clocks. Read the union node documentation and trace the union processing path to determine how timestamps affect queued points. Done means the missing `system` points are accounted for and the accepted clock-skew behavior is fixed or clearly documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100