spring-cloud / spring-cloud/spring-cloud-gateway
Noticing leaks in release of byte buffers
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 4.9k
- Forks
- 3.5k
- Avg merge
- 20h 57m
- Merged PRs (30d)
- 8
Description
Describe the bug
We are running an gateway application (has underlying Spring Cloud Gateway v4.0.6) that currently supports Spring Boot 3.1.x. Recently we have started to see issues of memory leaks during the handling of the responses in the filters of SCG that is impacting our application's traffic. We did try to narrow it down by disabling our custom filters and run with default SCG filters and we could still see leaks occurring when SCG is processing the response. (a partial snippet added below).
{"instant":{"epochSecond":1715881307,"nanoOfSecond":613266084},"thread":"egress-h2c-epoll-4","level":"ERROR","loggerName":"io.netty.util.ResourceLeakDetector","message":"LEAK:
ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more
information.\nRecent access records:
\n#1:\n\tio.netty.handler.codec.http.DefaultHttpContent.release(DefaultHttpContent.java:92)\n\tio.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:90)\n\treactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:296)\n\treactor.netty.channel.FluxReceive.request(FluxReceive.java:131)\n\treactor.core.publisher.FluxMap$MapSubscriber.request(FluxMap.java:164)\n\treactor.core.publisher.FluxPeek$PeekSubscriber.request(FluxPeek.java:138)\n\treactor.core.publisher.FluxMap$MapSubscriber.request(FluxMap.java:164)\n\treactor.core.publisher.Operators$BaseFluxToMonoOperator.request(Operators.java:2041)
The above log is printed when we include the jvm argument -Dio.netty.leakDetection.level=paranoid
Springboot Version: 3.1.5
Spring Cloud Gateway Version: 4.0.6
Netty Version: 4.1.100
Jetty Version: 11.0.17
This can be easily reproduced when we have RetryGatewayFilter is enabled.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the leak with RetryGatewayFilter enabled and -Dio.netty.leakDetection.level=paranoid, using the reported Spring Boot, Spring Cloud Gateway, Netty, and Jetty versions. Trace response handling through the gateway filters and compare the leak records with the RetryGatewayFilter path. Done means the reproducible response flow no longer reports unreleased ByteBuf instances.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spring, spring-boot
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100