apache / apache/incubator-seata
When destroying `RmNettyRemotingClient`, all `TableMetaRefreshHolder` should actively close the process
- Dominant language
- Java
- Stars
- 26k
- Forks
- 8.8k
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 4
Description
### Check Ahead
- [x] I have searched the [issues](https://github.com/seata/seata/issues) of this repository and believe that this is not a duplicate.
- [ ] I am willing to try to fix this bug myself.
### Ⅰ. Issue Description
- This issue is actually the successor of https://github.com/apache/incubator-seata/issues/7042 , but because https://github.com/apache/incubator-seata/pull/7044 circumvents the problems encountered when using HikariCP in a clever way, I cannot reproduce the problem described in the current issue using only Postgres JDBC Driver and Seata Client in https://github.com/linghengqian/seata-refresh-holder-test/blob/master/src/test/java/com/github/linghengqian/SimpleTest.java .🤣
- To explain what I found in https://github.com/apache/shardingsphere/pull/35427 , I introduced a minimal reproducible unit test in https://github.com/linghengqian/seata-refresh-holder-test/blob/master/src/test/java/com/github/linghengqian/ShardingSphereTest.java . This needs to be explained by introducing the ShardingSphere Proxy into the process.
- When shardingsphere proxy is started in-process and Seata Client is destroyed by `org.apache.shardingsphere.mode.manager.ContextManager#close()`, the following operation of `org.apache.shardingsphere.transaction.base.seata.at.SeataATShardingSphereTransactionManager#close()` is not enough to completely destroy Seata Client.
```java
import org.apache.seata.config.ConfigurationFactory;
import org.apache.seata.core.rpc.netty.RmNettyRemotingClient;
import org.apache.seata.core.rpc.netty.TmNettyRemotingClient;
public void close() {
TmNettyRemotingClient.getInstance().destroy();
RmNettyRemotingClient.getInstance().destroy();
ConfigurationFactory.reload();
}
```
- Once I execute `Awaitility.await().timeout(1L, TimeUnit.HOURS).pollDelay(2L, TimeUnit.MINUTES).until(() -> true);` in https://github.com/linghengqian/seata-refresh-holder-test/blob/master/src/test/java/com/github/linghengqian/ShardingSphereTest.java, the internal thread of Seata Client will continue to try to refresh the metadata.
```
[INFO] Running com.github.linghengqian.ShardingSphereTest
[ERROR] 2025-07-11 16:05:07.688 [ForkJoinPool.commonPool-worker-1] o.a.s.config.ConfigurationFactory - failed to load non-spring configuration :not found service provider for : org.apache.seata.config.ConfigurationProvider
org.apache.seata.common.loader.EnhancedServiceNotFoundException: not found service provider for : org.apache.seata.config.ConfigurationProvider
[ERROR] 2025-07-11 16:05:09.252 [Connection-2-ThreadExecutor] o.a.s.config.ConfigurationFactory - failed to load non-spring configuration :not found service provider for : org.apache.seata.config.ConfigurationProvider
org.apache.seata.common.loader.EnhancedServiceNotFoundException: not found service provider for : org.apache.seata.config.ConfigurationProvider
[ERROR] 2025-07-11 16:05:09.516 [Connection-3-ThreadExecutor] o.a.s.config.ConfigurationFactory - failed to load non-spring configuration :not found service provider for : org.apache.seata.config.ConfigurationProvider
org.apache.seata.common.loader.EnhancedServiceNotFoundException: not found service provider for : org.apache.seata.config.ConfigurationProvider
[ERROR] 2025-07-11 16:05:16.683 [main] o.a.s.config.ConfigurationFactory - failed to load non-spring configuration :not found service provider for : org.apache.seata.config.ConfigurationProvider
org.apache.seata.common.loader.EnhancedServiceNotFoundException: not found service provider for : org.apache.seata.config.ConfigurationProvider
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 137.6 s <<< FAILURE! -- in com.github.linghengqian.ShardingSphereTest
[ERROR] com.github.linghengqian.ShardingSphereTest.test -- Time elapsed: 137.4 s <<< ERROR!
java.lang.NullPointerException: Cannot read field "tableMetaRefreshQueue" because "x0" is null
at org.apache.seata.rm.datasource.sql.struct.TableMetaCacheFactory$TableMetaRefreshHolder.access$000(TableMetaCacheFactory.java:112)
at org.apache.seata.rm.datasource.sql.struct.TableMetaCacheFactory.tableMetaRefreshEvent(TableMetaCacheFactory.java:98)
at org.apache.seata.rm.datasource.sql.struct.TableMetaCacheFactory$TableMetaRefreshHolder.lambda$new$0(TableMetaCacheFactory.java:131)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:1570)
```
- `org.apache.seata.rm.datasource.sql.struct.TableMetaCacheFactory` has no public Java API or associated Java API to cancel all executing `TableMetaRefreshHolder`. The only judgment criterion is the error code `08006` thrown when obtaining the data source. This caused the unit tests to fail and terminate execution.
- I would say this is obviously related to shardingsphere not throwing a standard error code, but I did close all internal HikariCP data sources in the relevant functions of https://github.com/linghengqian/seata-refresh-holder-test/blob/master/src/test/java/com/github/linghengqian/ProxyTestingServer.java used in the unit test.
```java
public void close(final List logicDataBaseNameList) {
ContextManager contextManager = ProxyContext.getInstance().getContextManager();
logicDataBaseNameList.forEach(logicDataBaseName -> contextManager.getStorageUnits(logicDataBaseName)
.values()
.stream()
.map(StorageUnit::getDataSource)
.forEach(dataSource -> {
if (dataSource instanceof AutoCloseable) {
try {
((AutoCloseable) dataSource).close();
} catch (Exception ex) {
throw new RuntimeException(ex);
}
}
}));
contextManager.close();
completableFuture.cancel(false);
Awaitility.await().atMost(1L, TimeUnit.MINUTES).until(completableFuture::isDone);
}
```
- It's hard to say whether this is solved from within shardingsphere or from the seata client side. Shardingsphere actually hasn't started using the seata client's internal Java API yet.
### Ⅱ. Describe what happened
- As mentioned above.
### Ⅲ. Describe what you expected to happen
- When destroying `org.apache.seata.core.rpc.netty.RmNettyRemotingClient`, all `org.apache.seata.rm.datasource.sql.struct.TableMetaRefreshHolder` should actively close the process.
- Or at least provide a public Java API to destroy all `org.apache.seata.rm.datasource.sql.struct.TableMetaRefreshHolder` in the background.
### Ⅳ. How to reproduce it (as minimally and precisely as possible)
- I described how to start this unit test under Windows 11 Home 24H2 at https://github.com/linghengqian/seata-refresh-holder-test/blob/master/README.md . See https://github.com/linghengqian/seata-refresh-holder-test .
- Execute the following command on the `Windows 11 24H2` instance with `PowerShell/PowerShell`,
`version-fox/vfox`, `git-for-windows/git` and `rancher-sandbox/rancher-desktop` installed.
```shell
vfox add java
vfox install java@21.0.7-ms
vfox use --global java@21.0.7-ms
git clone git@github.com:linghengqian/seata-refresh-holder-test.git
cd ./seata-refresh-holder-test/
./mvnw -T 1C "-Dtest=ShardingSphereTest" clean test
```
- I also wrote the CI file under Ubuntu at https://github.com/linghengqian/seata-refresh-holder-test/blob/master/.github/workflows/test.yml, which can also be reproduced under Ubuntu. See https://github.com/linghengqian/seata-refresh-holder-test/actions/runs/16215169307/job/45782834023 .
-
### Ⅴ. Anything else we need to know?
- Unfortunately, without the introduction of ShardingSphere Proxy, this strange error log cannot be reproduced at all. Any help is greatly appreciated.
- And I still need to use `org.awaitility.Awaitility` to wait for a longer time than `client.rm.tableMetaCheckerInterval` to reproduce this bug.
### Ⅵ. Environment
- JDK: `Microsoft OpenJDK For JDK 21.0.7`
- Seata Server and Seata Client: `2.4.0`
- Database: Docker Image `postgres:17.2-bookworm`
- OS: `Windows 11 Home 24H2` and `Ubuntu 24.04`
Contributor guide
Assessment
This issue has not been assessed yet.