spring-cloud / spring-cloud/spring-cloud-gateway

Noticing leaks in release of byte buffers

Open
#3,399 11 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feedback-provided
Dominant language
Java
Stars
4.9k
Forks
3.5k
Avg merge
20h 57m
Merged PRs (30d)
8

Description

Describe the bug
We are running an gateway application (has underlying Spring Cloud Gateway v4.0.6) that currently supports Spring Boot 3.1.x. Recently we have started to see issues of memory leaks during the handling of the responses in the filters of SCG that is impacting our application's traffic. We did try to narrow it down by disabling our custom filters and run with default SCG filters and we could still see leaks occurring when SCG is processing the response. (a partial snippet added below).

{"instant":{"epochSecond":1715881307,"nanoOfSecond":613266084},"thread":"egress-h2c-epoll-4","level":"ERROR","loggerName":"io.netty.util.ResourceLeakDetector","message":"LEAK:
ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more
information.\nRecent access records:
\n#1:\n\tio.netty.handler.codec.http.DefaultHttpContent.release(DefaultHttpContent.java:92)\n\tio.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:90)\n\treactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:296)\n\treactor.netty.channel.FluxReceive.request(FluxReceive.java:131)\n\treactor.core.publisher.FluxMap$MapSubscriber.request(FluxMap.java:164)\n\treactor.core.publisher.FluxPeek$PeekSubscriber.request(FluxPeek.java:138)\n\treactor.core.publisher.FluxMap$MapSubscriber.request(FluxMap.java:164)\n\treactor.core.publisher.Operators$BaseFluxToMonoOperator.request(Operators.java:2041)

The above log is printed when we include the jvm argument -Dio.netty.leakDetection.level=paranoid

Springboot Version: 3.1.5
Spring Cloud Gateway Version: 4.0.6
Netty Version: 4.1.100
Jetty Version: 11.0.17

This can be easily reproduced when we have RetryGatewayFilter is enabled.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the leak with RetryGatewayFilter enabled and -Dio.netty.leakDetection.level=paranoid, using the reported Spring Boot, Spring Cloud Gateway, Netty, and Jetty versions. Trace response handling through the gateway filters and compare the leak records with the RetryGatewayFilter path. Done means the reproducible response flow no longer reports unreleased ByteBuf instances.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spring, spring-boot
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.