apache / apache/pekko

Reduce memory usage when large number of GraphStageLogic involves

Open
#1,624 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
1.6k
Forks
211
Avg merge
1d 6h
Merged PRs (30d)
89

Description

@queimadus's report in https://github.com/apache/pekko/discussions/1566
refs: https://github.com/apache/pekko/pull/1623

and with the code below
```scala
import org.apache.pekko.actor.ActorSystem
import org.apache.pekko.stream.scaladsl.{Sink, Source}
import org.apache.pekko.util.ByteString

import scala.concurrent.Await

object PekkoQuickstart extends App {
private implicit val system: ActorSystem = ActorSystem()

val s = Source
.repeat(())
.map(_ => ByteString('a' * 400000))
.take(1000000)
.prefixAndTail(50000)
.flatMapConcat { case (prefix, tail) => Source(prefix).concatLazy(tail) }

val r = Source.empty
.concatAllLazy(List.tabulate(30000)(_ => s): _*)
.runWith(Sink.ignore)

Await.result(r, scala.concurrent.duration.Duration.Inf)
println(r.value)

// Source
// .repeat(s)
// .take(30000)
// .flatMapConcat(x => x)
// .runWith(Sink.ignore)
// .onComplete(println(_))

// Source.empty
// .concatAllLazy(List.tabulate(30000)(_ => Source.lazySource(() => s)): _*)
// .runWith(Sink.ignore).onComplete(println(_))
}

```
image

we can get a heap dump of
![image](https://github.com/user-attachments/assets/4a5860c6-a43a-449d-99ff-901a17a5ac59)

image

To fix the problem, I think we need to clean the logic once we are done with a sub-graph, but the current code needs to get a snapshot for the materializer

To fix the problem I think we need to recycle the graph stage logics, which means more changes need to be done in the interpreter.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.