[Bug] In high-concurrency scenarios, the connection is interrupted abnormally
- Dominant language
- Java
- Stars
- 41.6k
- Forks
- 26.4k
- Avg merge
- 15h 13m
- Merged PRs (30d)
- 4
Description
### Pre-check
- [X] I am sure that all the content I provide is in English.
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dubbo/issues?q=is%3Aissue) and found no similar issues.
### Apache Dubbo Component
Java SDK (apache/dubbo)
### Dubbo Version
3.2.14
### Steps to reproduce this issue
my test code
client code
```
package org.example.client;
import org.apache.dubbo.config.ApplicationConfig;
import org.apache.dubbo.config.ConsumerConfig;
import org.apache.dubbo.config.MethodConfig;
import org.apache.dubbo.config.ReferenceConfig;
import org.apache.dubbo.config.bootstrap.DubboBootstrap;
import org.apache.dubbo.config.bootstrap.builders.ReferenceBuilder;
import org.apache.dubbo.remoting.Constants;
import org.apache.dubbo.rpc.FutureContext;
import org.example.api.GreetingsService;
import java.io.IOException;
import java.util.concurrent.*;
public class Application {
private static final ExecutorService worker = Executors.newFixedThreadPool(1, (r) ->{
Thread thread = new Thread(r);
thread.setName("worker-" + thread.getId());
return thread;
} );
private static final ExecutorService printer = Executors.newFixedThreadPool(1, (r) ->{
Thread thread = new Thread(r);
thread.setName("printer-"+thread.getId());
return thread;
});
public static void main(String[] args) throws IOException, InterruptedException {
ApplicationConfig application = new ApplicationConfig();
application.setName("client");
int requests = args.length > 0 ? Integer.parseInt(args[0]) : 100000;
String ip = args.length > 1 ? args[1] : "localhost";
MethodConfig methodConfig = new MethodConfig();
methodConfig.setName("sayHi");
methodConfig.setAsync(true);
methodConfig.setTimeout(1000000);
MethodConfig resetMethodConfig = new MethodConfig();
resetMethodConfig.setName("reset");
resetMethodConfig.setAsync(false);
ConsumerConfig consumerConfig = new ConsumerConfig();
consumerConfig.setCorethreads(1);
consumerConfig.setQueues(10000000);
consumerConfig.setTimeout(1000000);
consumerConfig.setThreads(1);
ReferenceConfig reference =
ReferenceBuilder.newBuilder()
.interfaceClass(GreetingsService.class)
.url("tri://"+ip+":50052")
.addMethod( methodConfig )
.addMethod( resetMethodConfig )
.consumer(consumerConfig)
.appendParameter(Constants.CONNECT_TIMEOUT_KEY, "1000000")
.build();
DubboBootstrap.getInstance()
.application(application)
.reference(reference)
.start();
GreetingsService service = reference.get();
service.reset(); //同步调用,重置服务端的计数器为0
CountDownLatch latch = new CountDownLatch(requests); // 创建一个计数器,用来记录请求的次数
int[] count = new int[1];
long start = System.currentTimeMillis();
for (int i = 0; i < requests; i++) {
service.sayHi("dubbo"); //这里的返回值为空,请不要使用
CompletableFuture future = FutureContext.getContext().getCompletableFuture();
worker.submit(() -> {
try {
String s = future.get();
latch.countDown();
printer.submit(() -> {
// if( ++count[0] % 1000 == 0 ) {
System.out.println("Receive result ======> " + s);
// }
});
} catch (InterruptedException e) {
throw new RuntimeException(e);
} catch (ExecutionException e) {
throw new RuntimeException(e);
}
});
}
try {
latch.await();
}catch (InterruptedException e)
{
e.printStackTrace();
}
long end = System.currentTimeMillis();
long delta = end - start;
Thread.sleep(1000);
System.out.println("总计" + requests + " 次");
System.out.println("耗时" + delta + " 毫秒");
System.out.println("TPS: " + requests * 1000L /delta + " 次/秒");
System.exit(0);
}
}
```
server code
```
package org.example.provider;
import org.apache.dubbo.config.ApplicationConfig;
import org.apache.dubbo.config.bootstrap.DubboBootstrap;
import org.apache.dubbo.config.bootstrap.builders.ProtocolBuilder;
import org.apache.dubbo.config.bootstrap.builders.ServiceBuilder;
import org.example.api.GreetingsService;
public class Application {
public static void main(String[] args) {
ApplicationConfig application = new ApplicationConfig();
application.setName("provider");
DubboBootstrap.getInstance()
.application(application)
.protocol(ProtocolBuilder.newBuilder()
.dispatcher("all")
.accepts(1000000)
.queues(1000000)
.threadpool("fixed")
.threads(1)
.name("tri")
.port(50052)
.build())
.service(
ServiceBuilder.newBuilder()
.interfaceClass(GreetingsService.class)
.ref(new GreetingsServiceImpl())
.build())
.start()
.await();
}
}
```
api code
```
package org.example.api;
public interface GreetingsService {
String sayHi(String name);
String reset();
}
```
api impl code
```
package org.example.provider;
import org.example.api.GreetingsService;
public class GreetingsServiceImpl implements GreetingsService {
private int count = 0;
// private AtomicInteger count = new AtomicInteger(0);
@Override
public String sayHi(String name) {
return "hi, " + ++count + "次 " + name;
}
@Override
public String reset() {
count = 0;
return "reset success";
}
}
```
server 192.168.253.176 startup commad
`java -cp Dubbo3Test-1.0-SNAPSHOT.jar:./lib/* org.example.provider.Application -Djava.net.preferIPv4Stack=true -Dio.netty.leakDetectionLevel=advanced -Xmx2048m -Xms2048m`
client test startup command
`java -Djava.net.preferIPv4Stack=true -Ddubbo.application.qos-port=33333 -Ddubbo.application.qos-enable=false -Dio.netty.leakDetectionLevel=advanced -Xmx2048m -Xms2048m -cp Dubbo3Test-1.0-SNAPSHOT.jar:lib/* org.example.client.Application 100000 192.168.253.176`
The VM specifications for both the client and server are 2C4G
tcp settings
```
root@ubuntu01:/home/zwb# sysctl -p
net.ipv4.tcp_rmem = 4096 832256 13320192
net.core.rmem_default = 2129920
net.core.rmem_max = 13320192
net.ipv4.tcp_mem = 44379 208128 524288
net.core.wmem_max = 13320192
net.ipv4.tcp_wmem = 4096 832256 13320192
```
ERROR occurs on the client. Procedure
```
09:40:35.711 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker: -| [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions.
09:40:35.711 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker: -| [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions.
09:40:35.711 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker: -| [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions.
09:40:35.712 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker: -| [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions.
09:40:35.712 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker: -| [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions.
09:40:35.712 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker: -| [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions.
09:40:35.712 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker: -| [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions.
09:40:35.712 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker: -| [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions.
09:40:35.712 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker: -| [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions.
09:40:35.712 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker: -| [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions.
09:40:35.712 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker: -| [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions.
09:40:35.712 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker: -| [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions.
09:40:35.712 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker: -| [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions.
09:40:35.712 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker: -| [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions.
09:40:35.712 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker: -| [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions.
```
WARN occurs on the server. Procedure
```
09:40:23.588 |-INFO [NettyServerWorker-5-1] ing.transport.netty4.NettyChannelHandler:56 -| [DUBBO] The connection of /192.168.253.201:44932 -> /192.168.253.176:50052 is established., dubbo version: 3.2.14, current host: 192.168.253.193
09:40:32.918 |-INFO [NettyServerWorker-5-1] ting.transport.netty4.NettyServerHandler:93 -| [DUBBO] The connection of /192.168.253.201:44932 -> /192.168.253.176:50052 is disconnected., dubbo version: 3.2.14, current host: 192.168.253.193
09:40:32.919 |-WARN [NettyServerWorker-5-1] .dubbo.remoting.transport.AbstractServer: -| [DUBBO] All clients has disconnected from /192.168.253.176:50052. You can graceful shutdown now., dubbo version: 3.2.14, current host: 192.168.253.193, error code: 99-0. This may be caused by unknown error in remoting module, go to https://dubbo.apache.org/faq/99/0 to find instructions.
09:40:32.925 |-INFO [NettyServerWorker-5-1] ing.transport.netty4.NettyChannelHandler:72 -| [DUBBO] The connection of /192.168.253.201:44932 -> /192.168.253.176:50052 is disconnected., dubbo version: 3.2.14, current host: 192.168.253.193
09:40:34.988 |-INFO [NettyServerWorker-5-2] ing.transport.netty4.NettyChannelHandler:56 -| [DUBBO] The connection of /192.168.253.201:36978 -> /192.168.253.176:50052 is established., dubbo version: 3.2.14, current host: 192.168.253.193
```
### What you expected to happen
The client program doesn't end as expected and keeps getting stuck
### Anything else
_No response_
### Are you willing to submit a pull request to fix on your own?
- [X] Yes I am willing to submit a pull request on my own!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Assessment
This issue has not been assessed yet.