grpc / grpc/grpc-java

InProcess: client streaming memory leak when retry enabled

Open
#8,712 10 comments 3 reactions 1 assignee Assigned to @shivaspeaks View on GitHub
bug enhancement
Dominant language
Java
Stars
12.1k
Forks
4k
Avg merge
2d 17h
Merged PRs (30d)
37

Description

### What version of gRPC-Java are you using?

1.42.0

### What is your environment?

Linux 5.15.2
openjdk version "11.0.13" 2021-10-19

### What did you expect to see?

No java heap space exception.

### What did you see instead?

`java.lang.OutOfMemoryError: Java heap space`

### Steps to reproduce the bug

We've recently updated our grpc dependency from 1.39.0 to 1.42.0 which now causes an out of memory error in one of our tests.

I've attached a test project that can reproduce the problem.
[grpcheap.zip](https://github.com/grpc/grpc-java/files/7562661/grpcheap.zip)

How to run:

```
mvn clean test -Dgrpc.version=1.39.0 # test runs successfully
mvn clean test -Dgrpc.version=1.42.0 # test fails with OOM
```

Please be aware that the test is run with limited heap size (see pom.xml): `-Xmx64M`

The general idea of the test is this:
Client-side has an `InputStream` that produces ~200mb data. This data of that stream is (lazily) split into ~1mb chunks and these are then streamed to the server.

What happens when I run the test with grpc >1.40 is that I get an OOM when it has sent ~60mb worth of chunks, which obviously is near the `-Xmx` limit. However, what I would expect is that the heap usage would never rise to that level, as after each 1mb chunk has been sent that chunk can be gc'ed. I've also attached a screenshot of visualvm where you can see the heap graph of the test using grpc 1.39.0 vs 1.42.0:

![grpc](https://user-images.githubusercontent.com/574849/142431013-583ee74c-6332-4991-95fd-822f2ffba3b8.png)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.