Piped streams not flowing with batch
- Dominant language
- JavaScript
- Stars
- 3.4k
- Forks
- 145
- PR merge metrics
- No merged PRs in 30d
Description
We're dealing with an issue where piping streams through a highland batched stream causes it to halt. The actual code involves streaming data from HTTP, but I think I've managed to narrow it down to the following code.
```
let {Transform, Writable} = require('stream');
let _ = require('highland');
// Dummy transform
class NullTransform extends Transform {
constructor(options) {
options = options || {};
options.writableObjectMode = true;
options.readableObjectMode = true;
super(options);
}
_transform(chunk, encoding, callback) {
console.log('transform', typeof chunk);
this.push(chunk);
callback();
}
_flush(callback) {
callback();
}
}
// Dummy writer
class NullWriter extends Writable {
constructor(options) {
options = options || {};
options.objectMode = true;
// options.highWaterMark = 1;
super(options);
this._storage = [];
}
_write(chunk, encoding, callback) {
console.log('write', chunk, typeof chunk, chunk.length);
callback();
}
}
// Infinite stream of random numbers
const readStream = _({
next() {
console.log('next!');
return {
done: false,
value: Math.random(),
};
}});
const transform = new NullTransform();
const writer = new NullWriter();
```
This will cause the application to exit:
```
readStream
.pipe(_().batch(1000))
.pipe(writer);
```
Strangely, adding the dummy transform before the batching causes it to flow properly:
```
readStream
.pipe(transform)
.pipe(_().batch(1000))
.pipe(writer);
```
Am I missing something?
Contributor guide
Research direction
Reproduce the issue with the minimal NullTransform, NullWriter, and readStream example, first comparing the two shown pipe chains. Then inspect Highland's batch stream entry point and its interaction with Node.js piping; done means the direct batched pipeline continues flowing rather than exiting while the transform variant still behaves consistently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js
- Domain
- backend, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100