Proposal Stream.buffer(highWaterMark)
- Dominant language
- JavaScript
- Stars
- 3.4k
- Forks
- 145
- PR merge metrics
- No merged PRs in 30d
Description
I propose Stream.buffer as a new method that mimics the functionality of the node stream writev function
```
const deleteKeys = new Writable({
highWaterMark: 50,
write: (key, encoding, callback) => {
//handle 1 item in stream
},
writev: (keys, callback) => {
//keys contains all keys that have been buffered since the last iteration started
},
});
```
This fills a similar role to batch. the difference is that batch waits for the provided count to be pushed to it before it pushes the array downstream.
buffer would collect items until downstream requests another batch up to a limit
```
_(source)
.buffer(50)
.tap(batch => {
//batch contains all items that have flown downstream while tap was processing its last iteration (limit 50)
});
```
I am willing to write this assuming it is something the project is willing to accept.
Contributor guide
Research direction
The issue names no files or tests. Begin by reviewing the existing batch operator and Node's writev/highWaterMark behavior; agreement on the API semantics and a regression test demonstrating the proposed downstream buffering behavior would define done.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript
- Domain
- stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100