Unexpected parallelism degradation after merging streams with parEvalX
Nobody has claimed this yet.
- Dominant language
- Scala
- Stars
- 2.5k
- Forks
- 636
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 7
Description
While the behavior described here seems weird to me, I may miss something obvious. Excuse me if this is the case.
So I have two streams and each does a parEvalMap(8). Then I merge those 2 streams and call parEvalMap(8) again after the merge. Weirdly enough, parallelism on that last parEvalMap drops to 2.
If one of the input streams is empty, it's even worse - parallelism drops to 1.
It works the same with any kind of parEval... processing, not only parEvalMap.
Problem only happens if there's parEval... happening on the input streams. Without these merge works fine.
Using parJoinUnbounded instead of merge resolves the issue - parallelism stays intact both before and after the merge.
Here's a simple reproduction:
https://scastie.scala-lang.org/FkBB6WnHSESF9qXupHBNcg
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked Scastie reproduction and compare the merged streams with the parJoinUnbounded variant. Trace the merge behavior when both input streams use parEvalMap, including the case where one input is empty. Done means the downstream parEval processing preserves the expected parallelism in both cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100