influxdata / influxdata/kapacitor
UnionNode still buffering for no good reason
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
So, like #1100 it seems I'm running a foul of UnionNode.
I was trying to do something to the effect of:
var incoming = stream|from()
var group1 = incoming|where()
var group2 = incoming|where()
var group3 = incoming|where()
var group4 = incoming|where()
var group5 = incoming|where()
var group6 = incoming|where()
group1|union(group2,group3,group4,group5,group6)|influxDBOut()
I was consistently seeing the influx node running at least 30 points behind the number of points it should have received from the wheres. Watching it on stats I could see the node would "jump" every 30 seconds or so while the rest of the graph was processing at a fairly consistent 1 point per second.
To test the theory that it was Union, I've copied the influx lines on to the end of all the blocks, so now I have something more like this:
var incoming = stream|from()
incoming|where()|influxDBOut()
incoming|where()|influxDBOut()
incoming|where()|influxDBOut()
incoming|where()|influxDBOut()
incoming|where()|influxDBOut()
incoming|where()|influxDBOut()
And now I have it keeping perfect time at just a few milliseconds latency.
Any ideas? I'm running Kapacitor 1.3.1 so I thought the Union buffering was fixed now, since changelog says the buffering fix went in to 1.2.0.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the two Kapacitor 1.3.1 stream topologies described in the issue, comparing UnionNode followed by influxDBOut() with separate influxDBOut() nodes. Inspect the UnionNode behavior and the buffering fix referenced in the 1.2.0 changelog; done means the union path keeps pace without periodic 30-second jumps.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100