alibaba / alibaba/arthas

arthas使用netty的ByteBuf没有释放导致netty堆外内存泄漏

Open
#2,834 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
37.5k
Forks
7.6k
Avg merge
1d 22h
Merged PRs (30d)
4

Description

- [ ] 我已经在 [issues](https://github.com/alibaba/arthas/issues) 里搜索,没有重复的issue。

### 环境信息

* `arthas-boot.jar` 或者 `as.sh` 的版本: 3.7.2
* Arthas 版本: 3.7.2
* 操作系统版本: linux
* 目标进程的JVM版本: jdk8
* 执行`arthas-boot`的版本: 3.7.2

### 重现问题的步骤

1. Base64Command 使用netty的ByteBuf没有释放导致netty堆外内存泄漏
2. TunnelClientSocketClientHandler 使用netty的ByteBuf没有释放导致netty堆外内存泄漏

### 期望的结果

What do you expected from the above steps?

### 实际运行的结果

实际运行结果,最好有详细的日志,异常栈。尽量贴文本。

```
2024-05-14 09:59:12 [arthas-NettyWebsocketTtyBootstrap-5-16] ERROR c.a.a.d.i.n.u.ResourceLeakDetector -LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records:
Created at:
com.alibaba.arthas.deps.io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:96)
com.alibaba.arthas.deps.io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
com.alibaba.arthas.deps.io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179)
com.alibaba.arthas.deps.io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:116)
com.alibaba.arthas.deps.io.netty.handler.codec.base64.Base64.encode(Base64.java:108)
com.alibaba.arthas.deps.io.netty.handler.codec.base64.Base64.encode(Base64.java:100)
com.alibaba.arthas.deps.io.netty.handler.codec.base64.Base64.encode(Base64.java:80)
com.alibaba.arthas.deps.io.netty.handler.codec.base64.Base64.encode(Base64.java:70)
com.alibaba.arthas.deps.io.netty.handler.codec.base64.Base64.encode(Base64.java:66)
com.alibaba.arthas.tunnel.client.TunnelClientSocketClientHandler.channelRead0(TunnelClientSocketClientHandler.java:127)
com.alibaba.arthas.tunnel.client.TunnelClientSocketClientHandler.channelRead0(TunnelClientSocketClientHandler.java:33)
com.alibaba.arthas.deps.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
com.alibaba.arthas.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
com.alibaba.arthas.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
com.alibaba.arthas.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
com.alibaba.arthas.deps.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
com.alibaba.arthas.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
com.alibaba.arthas.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
com.alibaba.arthas.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
com.alibaba.arthas.deps.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
com.alibaba.arthas.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
com.alibaba.arthas.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
com.alibaba.arthas.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
com.alibaba.arthas.deps.io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
com.alibaba.arthas.deps.io.netty.handler.codec.http.websocketx.Utf8FrameValidator.channelRead(Utf8FrameValidator.java:89)
com.alibaba.arthas.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
com.alibaba.arthas.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
com.alibaba.arthas.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
com.alibaba.arthas.deps.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
com.alibaba.arthas.deps.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:333)
com.alibaba.arthas.deps.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:454)
com.alibaba.arthas.deps.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
com.alibaba.arthas.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
com.alibaba.arthas.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
com.alibaba.arthas.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
com.alibaba.arthas.deps.io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1383)
com.alibaba.arthas.deps.io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246)
com.alibaba.arthas.deps.io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295)
com.alibaba.arthas.deps.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
com.alibaba.arthas.deps.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
com.alibaba.arthas.deps.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
com.alibaba.arthas.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
com.alibaba.arthas.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
com.alibaba.arthas.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
com.alibaba.arthas.deps.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
com.alibaba.arthas.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
com.alibaba.arthas.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
com.alibaba.arthas.deps.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
com.alibaba.arthas.deps.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
com.alibaba.arthas.deps.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
com.alibaba.arthas.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
com.alibaba.arthas.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
com.alibaba.arthas.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
com.alibaba.arthas.deps.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
com.alibaba.arthas.deps.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
com.alibaba.arthas.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
```

Contributor guide

Open the contributing guide

Research direction

Read Base64Command and TunnelClientSocketClientHandler, especially channelRead0 at TunnelClientSocketClientHandler.java:127, and trace the ByteBuf ownership shown in the leak stack. Reproduce with the supplied Arthas 3.7.2, Linux, and JDK 8 setup, then verify both paths complete without ResourceLeakDetector warnings.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
networking, performance
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.