caolan / caolan/highland

Piped streams not flowing with batch

Open
#617 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
3.4k
Forks
145
PR merge metrics
No merged PRs in 30d

Description

We're dealing with an issue where piping streams through a highland batched stream causes it to halt. The actual code involves streaming data from HTTP, but I think I've managed to narrow it down to the following code.

```
let {Transform, Writable} = require('stream');
let _ = require('highland');

// Dummy transform
class NullTransform extends Transform {
constructor(options) {
options = options || {};
options.writableObjectMode = true;
options.readableObjectMode = true;
super(options);
}
_transform(chunk, encoding, callback) {
console.log('transform', typeof chunk);
this.push(chunk);
callback();
}
_flush(callback) {
callback();
}
}
// Dummy writer
class NullWriter extends Writable {
constructor(options) {
options = options || {};
options.objectMode = true;
// options.highWaterMark = 1;
super(options);
this._storage = [];
}

_write(chunk, encoding, callback) {
console.log('write', chunk, typeof chunk, chunk.length);
callback();
}
}

// Infinite stream of random numbers
const readStream = _({
next() {
console.log('next!');
return {
done: false,
value: Math.random(),
};
}});
const transform = new NullTransform();
const writer = new NullWriter();
```

This will cause the application to exit:
```
readStream
.pipe(_().batch(1000))
.pipe(writer);
```
Strangely, adding the dummy transform before the batching causes it to flow properly:
```
readStream
.pipe(transform)
.pipe(_().batch(1000))
.pipe(writer);
```

Am I missing something?

Contributor guide

Open the contributing guide

Research direction

Reproduce the issue with the minimal NullTransform, NullWriter, and readStream example, first comparing the two shown pipe chains. Then inspect Highland's batch stream entry point and its interaction with Node.js piping; done means the direct batched pipeline continues flowing rather than exiting while the transform variant still behaves consistently.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
backend, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.