dromara / dromara/dynamic-tp

[BUG]: use GrpcDtpAdapter occur OOM problem, DtpRunnable cannot be GC, stopWatchMap is always referenced

Open
#584 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
4.8k
Forks
860
PR merge metrics
No merged PRs in 30d

Description

### 平台

Linux

### Jdk版本

JDK21

### SpringBoot版本

3.4.1

### DynamicTp版本

1.1.9.1-3.x

### 配置中心类型

_No response_

### 配置中心版本

_No response_

### 问题描述

1. We initially used 1.1.9.1-3.x. After referencing dynamic-tp-spring-boot-starter-adapter-grpc, the grpc client reported an error when calling the grpc interface of other services.:rpc call result is null:INTERNAL: Panic! This is a bug! The reason for the error should be here: https://github.com/dromara/dynamic-tp/pull/520

Image

Image

2. Then we upgraded to version 1.2.x and looked at the source code of DTP. The above problem had been fixed when adapting to grpc, that is, the original thread pool would not be closed.

Image

3. However, because 1.2.x is a major version, the formats of some packages and DTP configuration files have also changed. In order to reduce the impact on the business layer, we still maintain the 1.1.9.1-3.x version, but the GrpcDtpAdapter of the 1.1.9.1-3.x version is changed to use the method after the 1.2.x fix, which is not to close the original thread pool.

Image

4. We have discovered another problem. The memory usage of the application will gradually increase.

Image
Let's go into the container and execute:jmap -histo:live 1 > live_histogram.txt,I saw that there was an object that was not dropped by GC and kept rising.

org.dromara.dynamictp.core.support.task.runnable.DtpRunnable

Image

We dumped the file and analyzed it with MAT, jmap -dump:live,format=b,file=heap.bin 1
It was found that this DtpRunnable has been referenced and cannot be recycled

Image

5. But we have seen here that the grpc thread will go to afterExecute when it completes the task, and the AwareManager.afterExecute inside will be removed, so I don’t know why. At present, as long as we are not suitable for grpc, there will be no problem if we delete GrpcDtpAdapter.

Image

Image

### 复现步骤

n version 1.1.9.1-3.x, using the GrpcDtpAdapter processing method in 1.2.x will cause OOM problems (we have not tried to use the 1.2.x version completely)

### 其他信息

_No response_

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.