apache / apache/openwhisk

Memory leak in `akka.actor.LocalActorRef`

Open
#5,431 14 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
6.8k
Forks
1.2k
Avg merge
2d 14h
Merged PRs (30d)
2

Description

# Summary

I'm working on upgrading OpenWhisk to Akka 2.6.20 and Scala 2.13 and experienced the issue with OpenWhisk invokers consuming all available g1-old heap size after running for a couple of days with active traffic.

Doing heap profiling I got the following suggestions from Heap Hero:
```
One instance of akka.actor.LocalActorRef loaded by jdk.internal.loader.ClassLoaders$AppClassLoader @ 0x3c05c5018
occupies 20,136,784 (18.14%) bytes.
The memory is accumulated in one instance of scala.collection.immutable.RedBlackTree$Tree,
loaded by jdk.internal.loader.ClassLoaders$AppClassLoader @ 0x3c05c5018, which occupies 20,132,728 (18.14%) bytes.
```

Further analysis with Eclipse Memory Analyzer shows the following:
Screenshot 2023-08-03 at 11 26 51 AM
Screenshot 2023-08-03 at 11 29 54 AM

![01](https://github.com/apache/openwhisk/assets/2736528/26c7875b-ccba-40c0-b5aa-982827a30e39)

## Environment details:

* Scala 2.13
* Akka 2.6.20
* Akka HTTP 10.2.10
* Akka Management 1.1.4

Any suggestions on where I should look to find the root cause of this memory leak?

Contributor guide

Open the contributing guide

Research direction

Start with the OpenWhisk invoker under active traffic using the stated Scala 2.13, Akka 2.6.20, Akka HTTP 10.2.10, and Akka Management 1.1.4 environment. Inspect the LocalActorRef and RedBlackTree retention shown by Heap Hero and Eclipse Memory Analyzer, then trace the relevant invoker and actor lifecycle code; done means identifying a reproducible leak cause and an agreed fix, but the issue names no source files or tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
backend, cloud
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
18/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.