async-rs / async-rs/parallel-stream

Move task spawning to driver methods

Open
#4 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
96
Forks
13
PR merge metrics
No merged PRs in 30d

Description

Currently parallel-stream will start doing work the second it's initialized. This is much the same as `task::spawn`. The upside of this is that we got it to work, and it's what people want in the overwhelming amount of cases.

The downsides are that we're both spawning more tasks than needed, which interferes with the ability of the compiler to inline futures, which in turn will impact performance. Also there's no backpressure.

The solution to this seems to be to move "task spawning" to the edge methods: `next`, `for_each`, `collect`, `sum` that on the one hand spawn tasks as fast as possible, while on the other hand allow outputting them one-by-one.

There's probably some nuance here; for example `next` is by nature sequential so task spawning might not even make sense. But overall it seems that if we can invert the logic slightly this could lead to some neat results.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.