caolan / caolan/highland

how to wait for async operations inside iterator of: Stream.reduce

Open
#707 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
3.4k
Forks
145
PR merge metrics
No merged PRs in 30d

Description

When consuming a stream with [`.reduce`](https://caolan.github.io/highland/#reduce), I'd like to be able to do some async operations inside the `iterator/reducer` function and have the stream wait for their completion before consuming next element.

Here's what I'm doing right now:

```js
const H = require('highland')
const Promise = require('aigle')

const reducer = async (memoPromise, x) => {
console.log('got x:', x)
const memo = await memoPromise
console.log('processing x:', x)
return Promise.delay(10).then(() => memo + x)
}

H([1, 2, 3])
.reduce(reducer, 0)
.toPromise(Promise)
.then(console.log)
```

However the stream is being consumed as fast as possible, without waiting for async operation inside the reducer to complete. The output of running the above is:
```
got x: 1
got x: 2
got x: 3
processing x: 1
processing x: 2
processing x: 3
6
```

while I'd like it to be:
```
got x: 1
processing x: 1
got x: 2
processing x: 2
got x: 3
processing x: 3
6
```

Contributor guide

Open the contributing guide

Research direction

Start with the Stream.reduce implementation and documentation, then reproduce the provided reducer example to observe when each input is consumed. Define done as reducer processing occurring sequentially, with the final result still resolving to 6 and the output matching the requested order.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
stream-processing
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.