influxdata / influxdata/kapacitor

union node not passing points

Open
#1,224 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

Script:
```
var system = stream|from().measurement('system')|log()
var ntpq = stream|from().measurement('ntpq')|log()
system|union(ntpq)|log()
```

Output:
```
[ntpq_orphan:log4] 2017/02/24 12:27:44 I! {"Name":"ntpq","Database":"telegraf","RetentionPolicy":"autogen","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{"env":"local","group":"uxdb","host":"influxdb","location":"infl","os":"Linux","refid":"10.1.103.20","remote":"10.0.2.12","stratum":"2"},"Fields":{"delay":3.464,"jitter":10.218,"offset":3.573275e+06,"poll":64,"reach":377,"when":38},"Time":"2017-02-24T12:27:38.782Z"}
[ntpq_orphan:log4] 2017/02/24 12:27:44 I! {"Name":"ntpq","Database":"telegraf","RetentionPolicy":"autogen","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{"env":"local","group":"uxdb","host":"influxdb","location":"infl","os":"Linux","refid":"10.1.103.20","remote":"10.0.2.12","stratum":"2"},"Fields":{"delay":3.464,"jitter":10.218,"offset":3.573275e+06,"poll":64,"reach":377,"when":43},"Time":"2017-02-24T12:27:43.817Z"}
[ntpq_orphan:log2] 2017/02/24 12:27:45 I! {"Name":"system","Database":"telegraf","RetentionPolicy":"autogen","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{"env":"local","host":"db1","location":"db1","os":"Linux"},"Fields":{"load1":0,"load15":0.05,"load5":0.01,"n_cpus":3,"n_users":0},"Time":"2017-02-24T14:01:50.131Z"}
[ntpq_orphan:log2] 2017/02/24 12:27:45 I! {"Name":"system","Database":"telegraf","RetentionPolicy":"autogen","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{"env":"local","host":"db1","location":"db1","os":"Linux"},"Fields":{"uptime":230904},"Time":"2017-02-24T14:01:50.131Z"}
[ntpq_orphan:log7] 2017/02/24 12:27:45 I! {"Name":"ntpq","Database":"telegraf","RetentionPolicy":"autogen","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{"env":"local","group":"uxdb","host":"influxdb","location":"infl","os":"Linux","refid":"10.1.103.20","remote":"10.0.2.12","stratum":"2"},"Fields":{"delay":3.464,"jitter":10.218,"offset":3.573275e+06,"poll":64,"reach":377,"when":28},"Time":"2017-02-24T12:27:28.802Z"}
[ntpq_orphan:log7] 2017/02/24 12:27:45 I! {"Name":"ntpq","Database":"telegraf","RetentionPolicy":"autogen","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{"env":"local","group":"uxdb","host":"influxdb","location":"infl","os":"Linux","refid":"10.1.103.20","remote":"10.0.2.12","stratum":"2"},"Fields":{"delay":3.464,"jitter":10.218,"offset":3.573275e+06,"poll":64,"reach":377,"when":33},"Time":"2017-02-24T12:27:33.782Z"}
[ntpq_orphan:log7] 2017/02/24 12:27:45 I! {"Name":"ntpq","Database":"telegraf","RetentionPolicy":"autogen","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{"env":"local","group":"uxdb","host":"influxdb","location":"infl","os":"Linux","refid":"10.1.103.20","remote":"10.0.2.12","stratum":"2"},"Fields":{"delay":3.464,"jitter":10.218,"offset":3.573275e+06,"poll":64,"reach":377,"when":38},"Time":"2017-02-24T12:27:38.782Z"}
[ntpq_orphan:log7] 2017/02/24 12:27:45 I! {"Name":"ntpq","Database":"telegraf","RetentionPolicy":"autogen","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{"env":"local","group":"uxdb","host":"influxdb","location":"infl","os":"Linux","refid":"10.1.103.20","remote":"10.0.2.12","stratum":"2"},"Fields":{"delay":3.464,"jitter":10.218,"offset":3.573275e+06,"poll":64,"reach":377,"when":43},"Time":"2017-02-24T12:27:43.817Z"}
[ntpq_orphan:log2] 2017/02/24 12:27:51 I! {"Name":"system","Database":"telegraf","RetentionPolicy":"autogen","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{"env":"local","host":"jump","location":"jump","os":"Linux"},"Fields":{"load1":0,"load15":0.05,"load5":0.01,"n_cpus":3,"n_users":0},"Time":"2017-02-23T13:18:58.856Z"}
[ntpq_orphan:log2] 2017/02/24 12:27:51 I! {"Name":"system","Database":"telegraf","RetentionPolicy":"autogen","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{"env":"local","host":"jump","location":"jump","os":"Linux"},"Fields":{"uptime":1622654},"Time":"2017-02-23T13:18:58.856Z"}
[ntpq_orphan:log2] 2017/02/24 12:27:54 I! {"Name":"system","Database":"telegraf","RetentionPolicy":"autogen","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{"env":"local","group":"uxdb","host":"influxdb","location":"infl","os":"Linux"},"Fields":{"load1":0.01,"load15":0.05,"load5":0.03,"n_cpus":3,"n_users":4},"Time":"2017-02-24T12:27:44.135Z"}
[ntpq_orphan:log2] 2017/02/24 12:27:54 I! {"Name":"system","Database":"telegraf","RetentionPolicy":"autogen","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{"env":"local","group":"uxdb","host":"influxdb","location":"infl","os":"Linux"},"Fields":{"uptime":1825019},"Time":"2017-02-24T12:27:44.135Z"}
[ntpq_orphan:log4] 2017/02/24 12:27:54 I! {"Name":"ntpq","Database":"telegraf","RetentionPolicy":"autogen","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{"env":"local","group":"uxdb","host":"influxdb","location":"infl","os":"Linux","refid":"10.1.103.20","remote":"10.0.2.12","stratum":"2"},"Fields":{"delay":3.464,"jitter":10.218,"offset":3.573275e+06,"poll":64,"reach":377,"when":48},"Time":"2017-02-24T12:27:48.784Z"}
[ntpq_orphan:log4] 2017/02/24 12:27:54 I! {"Name":"ntpq","Database":"telegraf","RetentionPolicy":"autogen","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{"env":"local","group":"uxdb","host":"influxdb","location":"infl","os":"Linux","refid":"10.1.103.20","remote":"10.0.2.12","stratum":"2"},"Fields":{"delay":3.464,"jitter":10.218,"offset":3.573275e+06,"poll":64,"reach":377,"when":53},"Time":"2017-02-24T12:27:53.781Z"}
[ntpq_orphan:log7] 2017/02/24 12:27:54 I! {"Name":"ntpq","Database":"telegraf","RetentionPolicy":"autogen","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{"env":"local","group":"uxdb","host":"influxdb","location":"infl","os":"Linux","refid":"10.1.103.20","remote":"10.0.2.12","stratum":"2"},"Fields":{"delay":3.464,"jitter":10.218,"offset":3.573275e+06,"poll":64,"reach":377,"when":48},"Time":"2017-02-24T12:27:48.784Z"}
[ntpq_orphan:log7] 2017/02/24 12:27:54 I! {"Name":"ntpq","Database":"telegraf","RetentionPolicy":"autogen","Group":"","Dimensions":{"ByName":false,"TagNames":null},"Tags":{"env":"local","group":"uxdb","host":"influxdb","location":"infl","os":"Linux","refid":"10.1.103.20","remote":"10.0.2.12","stratum":"2"},"Fields":{"delay":3.464,"jitter":10.218,"offset":3.573275e+06,"poll":64,"reach":377,"when":53},"Time":"2017-02-24T12:27:53.781Z"}
```

Notice that the only points passing to `log7` are the "ntpq" points. None of the "system" points are making it through.
However if I stop the task, all of a sudden all the "system" points are released.

This looks to be somehow related to the time. I found this issue in my development environment (bunch of vagrant VMs), and the clocks are horribly out of sync. Once I synced all the clocks it started behaving more normal. However the documentation on the union node gives no indication that it cares about time at all. And while my clocks were horribly out of sync, how out of sync do they have to be to break the union node? Will a few milliseconds break it?

This is kapacitor built from current master (d57e7ba).

Contributor guide

Open the contributing guide

Research direction

Start with the reported `system|union(ntpq)|log()` reproduction and compare behavior with synchronized and out-of-sync clocks. Read the union node documentation and trace the union processing path to determine how timestamps affect queued points. Done means the missing `system` points are accounted for and the accepted clock-skew behavior is fixed or clearly documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.