caolan / caolan/highland

Proposal Stream.buffer(highWaterMark)

Open
#655 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
3.4k
Forks
145
PR merge metrics
No merged PRs in 30d

Description

I propose Stream.buffer as a new method that mimics the functionality of the node stream writev function
```
const deleteKeys = new Writable({
highWaterMark: 50,
write: (key, encoding, callback) => {
//handle 1 item in stream
},
writev: (keys, callback) => {
//keys contains all keys that have been buffered since the last iteration started
},
});
```

This fills a similar role to batch. the difference is that batch waits for the provided count to be pushed to it before it pushes the array downstream.
buffer would collect items until downstream requests another batch up to a limit

```
_(source)
.buffer(50)
.tap(batch => {
//batch contains all items that have flown downstream while tap was processing its last iteration (limit 50)
});
```

I am willing to write this assuming it is something the project is willing to accept.

Contributor guide

Open the contributing guide

Research direction

The issue names no files or tests. Begin by reviewing the existing batch operator and Node's writev/highWaterMark behavior; agreement on the API semantics and a regression test demonstrating the proposed downstream buffering behavior would define done.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.