spring-projects / spring-projects/spring-framework
Direct Buffer not freed and Netty Resource Leaks
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 60.2k
- Forks
- 38.8k
- Avg merge
- 5d 2h
- Merged PRs (30d)
- 27
Description
First of all: I'm not sure if this is 100% the right place for this Issue as Netty, Reactor-Netty and Spring-Data mongodb are also involved.
After updating our Spring-Webflux app to 2.6.6 we encountered "java.lang.OutOfMemoryError Direct buffer memory" exceptions in our logs.
After some investigation I found out that the netty update to 4.1.75/4.1.76 might cause the problem.
When netty leak detection is set to paranoid we see the following leak report in our logs:
LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records:
#1:
io.netty.buffer.AdvancedLeakAwareByteBuf.writeBytes(AdvancedLeakAwareByteBuf.java:611)
org.springframework.core.io.buffer.NettyDataBuffer.write(NettyDataBuffer.java:177)
org.springframework.core.io.buffer.NettyDataBuffer.write(NettyDataBuffer.java:43)
org.springframework.http.codec.json.AbstractJackson2Encoder.encodeStreamingValue(AbstractJackson2Encoder.java:286)
org.springframework.http.codec.json.AbstractJackson2Encoder.lambda$encode$1(AbstractJackson2Encoder.java:168)
reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:106)
...
(complete one attached as file)
The use case behind is: Streaming pretty large data as json (~70kb per json) from mongodb to the client using a spring-webflux application.
Before the update to Spring-Boot 2.6.6. (and the including netty update) the leak was also present in the logs but did not make any problems so far. When starting the app on Java 11 it also seems that the buffer is cleaned fine (probably by GC) - but using Java 17 we always see the "java.lang.OutOfMemoryError Direct buffer memory" exceptions.
Our App runs on Cloudfoundry with MaxDirectMemorySize set to 80M which before the update was always enough. Increasins the MaxDirectMemorySize just delays the overflow exceptions.
Example to Reproduce the Issue:
Look at the readme of the repository for prerequisites and launch instructions.
For watching the direct buffer pools i used JVisualVM with the Buffer Pools plugins installed.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked webflux-buffer-overflow-demo repository and follow its README prerequisites and launch instructions. Inspect the attached leak log and the stack-trace entry points in NettyDataBuffer and AbstractJackson2Encoder while reproducing the streaming MongoDB response on Java 17. Done means the stated reproduction no longer produces the direct-buffer exhaustion or leak report.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mongodb, spring
- Domain
- backend, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100