typelevel / typelevel/fs2

Improve `parJoin` with the ability to select a join strategy

Open
#2,674 4 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature
Dominant language
Scala
Stars
2.5k
Forks
636
Avg merge
2d 4h
Merged PRs (30d)
7

Description

The current implementation of parJoin with maxOpen does not start the remaining streams until the earlier ones are complete. This is a problem in cases where the inner streams are infinite as in the example below:

Listing A - parJoin with 10 infinite streams, maxOpen = 2

    Stream.range(1, 10)
      .map {
        c =>
          Stream
            .repeatEval(IO(c))
      }
      .parJoin(maxOpen = 2)
      .evalTap {
        c =>
          IO(println(s"I am Stream $c"))
      }
      .take(10)
      .compile
      .drain

Output of Listing A

I am Stream 1
I am Stream 2
I am Stream 2
I am Stream 1
I am Stream 1
I am Stream 2
I am Stream 2
I am Stream 1
I am Stream 1
I am Stream 2

This limitation makes parJoin unusable with libraries like fs2-kafka when processing individual partitions is desirable before joining them. The only alternative in that case is to use parJoinUnbounded which fixes the above problem but has no way of controlling how many inner streams will be evaluated in parallel. This is desirable if the inner streams are memory intensive and you cannot let all of them run in parallel.

I would like to propose a new signature for parJoin as follows:


sealed trait JoinStrategy

object JoinStrategy {
  case object Random extends JoinStrategy
  case object RoundRobin extends JoinStrategy
  case object Sequential extends JoinStrategy
}

def parJoin(maxOpen: Int, joinStrategy: JoinStrategy = Sequential)

Sequential would be the current logic and the default
RoundRobin would evaluate each inner stream in turn
Random would be how the parJoinUnbounded works but with maximum parallel evaluations limited to maxOpen

Surly there can be better names selected for these and there could be other strategies but these are what I can think of now.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the parJoin and parJoinUnbounded entry points and compare their current scheduling behavior. Define how Sequential, RoundRobin, and Random should behave under maxOpen, including the default, then verify the selected strategy works for finite and infinite inner streams like the examples.】【。

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.