Serious Performance Issue With Pipeline
- Dominant language
- JavaScript
- Stars
- 3.4k
- Forks
- 145
- PR merge metrics
- No merged PRs in 30d
Description
I've been banging my head against this for the past few days after we got some pretty bad performance results following a recent load test. At first I thought it was a memory leak in our code but on further investigation I think the increase in memory usage was because `pipeline` was causing the CPU to max out so much that the garbage collector was fighting for CPU cycles. A stripped down test case showed the CPU hitting 99-100% for some pretty trivial stream processing with the memory staying constant.
Here's the test case:
``` javascript
var _ = require('highland');
var bigArray = []
for(var i = 0; i < 100000; i++) {
bigArray[i] = i;
}
var s = _(bigArray);
function addOne(a) {
return a + 1;
}
function bigAdd(a) {
for(var i = 0; i < 500; i++) {
a++;
}
return a;
}
function bigSubtract(a) {
for(var i = 0; i < 100; i++) {
a--;
}
return a;
}
console.time('timer');
s.through(_.pipeline(_.map(addOne), _.map(bigAdd), _.map(bigSubtract))).toArray(function () {
console.timeEnd('timer');
});
```

This takes nearly 40 seconds to run my machine whereas this only takes 142ms:
``` javascript
console.time('timer');
s.through(_.map(addOne)).through(_.map(bigAdd)).through(_.map(bigSubtract)).toArray(function () {
console.timeEnd('timer');
});
```
If I increase the size of `bigArray` to `1000000` then the first version takes so long that I have to kill it (compared to 1.3 seconds for the other version).
The problem seems to in the `wrapper` part of the `pipeline` function:
``` javascript
var wrapper = _(function (push, next) {
end.pull(function (err, x) {
if (err) {
wrapper._send(err);
next();
}
else if (x === nil) {
wrapper._send(null, nil);
}
else {
wrapper._send(null, x);
next();
}
});
});
wrapper.write = function (x) {
start.write(x);
};
```
The thing that's stumping me is that both `pull` and `send` really don't seem to do very much so I'm not sure how they can be leading to such a massive jump in CPU use. I want to take a stab at fixing this but I'll need a couple of pointers to get me started.
Contributor guide
Research direction
Start by running the two JavaScript reproductions in the issue and comparing their timings and CPU usage. Then inspect the shown wrapper in pipeline, tracing how pull, send, write, and next interact; done means pipeline preserves the chained transforms' results without the reported CPU and runtime spike.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js
- Domain
- backend, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100