apache / apache/uniffle

[Improvement] Add Netty buffer leak detection in integration test.

Open
#1,153 4 comments 0 reactions 1 assignee Claimed by @qijiale76 View on GitHub
Dominant language
Java
Stars
454
Forks
172
Avg merge
5d 17h
Merged PRs (30d)
5

Description

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.

### What would you like to be improved?

As mentioned in https://github.com/apache/incubator-uniffle/issues/1152, by adding `-Dio.netty.leakDetection.level=advanced`, we discovered the problem of buffer leak in Netty. It is necessary to include this leak detection feature in integration testing to better identify such problems.

```
[ERROR] 2023-08-16 12:56:11,453 Grpc-146 ResourceLeakDetector reportTracedLeak - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records:
#1:
io.netty.buffer.AdvancedLeakAwareByteBuf.order(AdvancedLeakAwareByteBuf.java:71)
io.netty.buffer.CompositeByteBuf.newComponent(CompositeByteBuf.java:346)
io.netty.buffer.CompositeByteBuf.consolidate0(CompositeByteBuf.java:1758)
io.netty.buffer.CompositeByteBuf.consolidateIfNeeded(CompositeByteBuf.java:571)
io.netty.buffer.CompositeByteBuf.addComponent(CompositeByteBuf.java:266)
io.netty.buffer.CompositeByteBuf.addComponent(CompositeByteBuf.java:222)
org.apache.uniffle.server.buffer.ShuffleBuffer.updateShuffleData(ShuffleBuffer.java:272)
org.apache.uniffle.server.buffer.ShuffleBuffer.getShuffleData(ShuffleBuffer.java:169)
org.apache.uniffle.server.buffer.ShuffleBufferManager.getShuffleData(ShuffleBufferManager.java:231)
org.apache.uniffle.server.ShuffleTaskManager.getInMemoryShuffleData(ShuffleTaskManager.java:516)
org.apache.uniffle.server.ShuffleServerGrpcService.getMemoryShuffleData(ShuffleServerGrpcService.java:798)
org.apache.uniffle.proto.ShuffleServerGrpc$MethodHandlers.invoke(ShuffleServerGrpc.java:1053)
io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:182)
io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35)
io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23)
io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:352)
io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:866)
io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java.base/java.lang.Thread.run(Thread.java:829)

......

: 14 leak records were discarded because they were duplicates
: 1010 leak records were discarded because the leak record count is targeted to 4. Use system property io.netty.leakDetection.targetRecords to increase the limit.

```

Ref: https://netty.io/wiki/reference-counted-objects.html

### How should we improve?

_No response_

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.