typelevel / typelevel/fs2

Unreliable stream interruption

Open
#2,330 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Scala
Stars
2.5k
Forks
636
Avg merge
2d 4h
Merged PRs (30d)
7

Description

When a stream is interrupted, I expect based on these docs that the interruption is final and can only be handled with Stream.bracket (or something built on Stream.bracket).

So, I would not expect the following code, which attempts to restart a stream indefinitely, to work when the provided stream is interrupted:

def resume[F[_] : Concurrent : RaiseThrowable, A, B](mk: A => Stream[F, B], checkpoint: B => A)(start: A): Stream[F, B] = {
  def go(s: Stream[F, Either[Throwable, B]], watermark: A): Pull[F, B, Unit] = s.pull.uncons1.flatMap {
    case Some((Right(b), rest)) => Pull.output1(b) >> go(rest, checkpoint(b))
    case Some((Left(_), _)) => go(mk(watermark).attempt, watermark)
    case None => go(mk(watermark).attempt, watermark)
  }

  go(mk(start).attempt, start).stream
}

However, I find that it sometimes works, and sometimes doesn't. In the following test:

// How many times should we interrupt the stream?
val Interrupts: Int = 1

// Interrupt the stream after five items, up to a max number of times
def interrupter[A](deferred: Deferred[IO, Unit], interruptCount: Ref[IO, Int]): Pipe[IO, A, A] = {
  input: Stream[IO, A] =>
    input.zipWithIndex
      .evalTap {
        case (_, 5) => interruptCount.getAndUpdate(_ + 1).flatMap { i =>
          if (i < Interrupts) {
            deferred.complete(())
          } else IO.unit
        }
        case _ => IO.unit
      }
      .map(_._1)
      .interruptWhen(deferred.get.attempt)
}

val stream: Int => Stream[IO, Int] = Stream.iterate(_)(_ + 1)

val assertion = for {
  interruptCount <- Ref.of[IO, Int](0)
  _ <- resume[Int, Int](
    start => Stream.eval(Deferred[IO, Unit]).flatMap(d => stream(start).through(interrupter(d, interruptCount))),
    _ + 1
  )(0)
    .take(1000)
    .compile
    .toList
    .map(lst => assert(lst == List.range(0, 1000))
} yield ()

if I only interrupt the stream a small number of times (e.g. once), most likely the restart function works and I get the 1000 elements from .take. But if I allow many interruptions (e.g. 10), typically the stream I get is truncated.

This is very surprising! Is this nondeterminism a bug?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Use the linked interruption documentation and the provided resume/interrupter reproduction as entry points. Run the assertion with one and many interruptions, then trace interruption handling to determine whether truncation is expected or a bug. Done means the nondeterminism is explained and the behavior is covered by the assertion or a focused regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.