typelevel / typelevel/fs2

Semantics of `fold` apparently broken by `observe` in case of failure

Open
#1,613 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Scala
Stars
2.5k
Forks
636
Avg merge
2d 4h
Merged PRs (30d)
7

Description

Hi.

The following piece of code shows that in version 1.0.5, when a pipe given to observe fails, a folded stream fails to comply with its guarantee of single emission.

import cats.effect.{ExitCode, IO, IOApp}
import cats.implicits._
import fs2.Stream

object TestApp extends IOApp {
  def run(args: List[String]): IO[ExitCode] =
    stream.compile.drain.as(ExitCode.Success)

  private def stream =
    for {
      count <- subStream
      _ <- Stream.eval(IO(println(s"Count: $count")))
    } yield ()

  private def subStream =
    Stream.range[IO](1, 10)
      .observe(_ => Stream.raiseError[IO](new RuntimeException("Just for fun")))
      .fold(0)((c, _) => c + 1)
}

The result is the following.

Count: 0
java.lang.RuntimeException: Just for fun
	at com.softwaymedical.itodirectory.extraction.TestApp$.$anonfun$subStream$1(TestApp.scala:19)
	at com.softwaymedical.itodirectory.extraction.TestApp$.$anonfun$subStream$1$adapted(TestApp.scala:19)
	at fs2.Stream$.through$extension(Stream.scala:2521)
	at fs2.Stream$InvariantOps$.sinkStream$1(Stream.scala:3493)
	...

Since fold should emit a single value, I expect the stream to fail without emitting anything. This is indeed the observed behavior when the implementation of subStream is replaced with:

Stream.raiseError[IO](new RuntimeException("Just for fun")).fold(0)((c, _) => c + 1)

Instead, a value is emitted, and then the streams fails.

As a result, the later effects (materialized here with Stream.eval(IO(println(s"Count: $count")))) take place despite the failure, which in my case leads to dramatic consequences.

I am aware of the concerns expressed in #1206 and #1208 and I was wondering whether they could be the cause.

I use observe a lot, mainly because I often need to use fold to gather information at the end of an ingestion process and use it afterwards. It forces me to shift my main logic into a "secondary" pipe given to observe, the reporting becoming the main stream so that I can preserve the result.

By the way, if there exists a better way to implement this kind of behavior, I would be glad to know about it.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the supplied minimal reproducer and inspect the semantics of Stream.observe and fold, including the related concerns in issues #1206 and #1208. Compare the failing observed stream with the direct Stream.raiseError case. Done means a failing observed pipe does not cause fold to emit a value before the stream fails, with regression coverage for the reported behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.