apache / apache/dubbo

[Bug] Dubbo 3.3.6 trip protocol netty memory leak

Open
#16,415 2 comments 0 reactions 0 assignees View on GitHub
help wanted
Dominant language
Java
Stars
41.6k
Forks
26.4k
Avg merge
15h 13m
Merged PRs (30d)
4

Description

### Pre-check

- [x] I am sure that all the content I provide is in English.

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/dubbo/issues?q=is%3Aissue) and found no similar issues.

### Apache Dubbo Component

Java SDK (apache/dubbo)

### Dubbo Version

dubbo-rpc-triple:3.3.6
Netty:4.2.9 use direct memory
jdk:17.0.17
windowns

### Steps to reproduce this issue

@DubboReference(
version = "1.0.0",
url = "tri://${dubbo.directUrl}",
protocol = "tri",
check = false,
timeout = 100, // Set a shorter timeout to simulate a rpc timeout
connections = 5,
actives = 5
)
public void setTripleService(TripleExternalService tripleService) {
this.tripleService = tripleService;
}

note : add jvm property -Dio.netty.leakDetectionLevel=PARANOID
If you keep calling a certain method of this interface for a while, it will appear, and the smaller the timeout, the easier it is to reproduce.
output:

io.netty.buffer.AdvancedLeakAwareByteBuf.readByte(AdvancedLeakAwareByteBuf.java:401)
io.netty.handler.codec.http2.DefaultHttp2FrameReader.preProcessFrame(DefaultHttp2FrameReader.java:190)
io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:146)
io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:39)
io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:186)
io.netty.handler.codec.http2.DecoratingHttp2ConnectionDecoder.decodeFrame(DecoratingHttp2ConnectionDecoder.java:61)
io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:391)
io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:451)
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:545)
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:484)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:356)
io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:293)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:354)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1429)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:918)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:172)
io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.handle(AbstractNioChannel.java:445)
io.netty.channel.nio.NioIoHandler$DefaultNioRegistration.handle(NioIoHandler.java:388)
io.netty.channel.nio.NioIoHandler.processSelectedKey(NioIoHandler.java:596)
io.netty.channel.nio.NioIoHandler.processSelectedKeysOptimized(NioIoHandler.java:571)
io.netty.channel.nio.NioIoHandler.processSelectedKeys(NioIoHandler.java:512)
io.netty.channel.nio.NioIoHandler.run(NioIoHandler.java:484)
io.netty.channel.SingleThreadIoEventLoop.runIo(SingleThreadIoEventLoop.java:225)
io.netty.channel.SingleThreadIoEventLoop.run(SingleThreadIoEventLoop.java:196)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:1195)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.base/java.lang.Thread.run(Thread.java:842)

### What you expected to happen

I don't know if I used it wrong or if it's actually a bug

### Anything else

_No response_

### Do you have a (mini) reproduction demo?

- [ ] Yes, I have a minimal reproduction demo to help resolve this issue more effectively!

### Are you willing to submit a pull request to fix on your own?

- [ ] Yes I am willing to submit a pull request on my own!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start by running the shown dubbo-rpc-triple 3.3.6 setup with Netty leak detection set to PARANOID, repeatedly calling the interface with the short timeout. Trace the reported Netty HTTP/2 stack and compare behavior as the timeout changes; done means the reproduction no longer reports a direct-memory leak and the cause is confirmed.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems, networking, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.