apache / apache/dubbo

[Bug] In high-concurrency scenarios, the connection is interrupted abnormally

Open
#14,445 5 comments 0 reactions 0 assignees View on GitHub
component/need-triage type/need-triage
Dominant language
Java
Stars
41.6k
Forks
26.4k
Avg merge
15h 13m
Merged PRs (30d)
4

Description

### Pre-check

- [X] I am sure that all the content I provide is in English.

### Search before asking

- [X] I had searched in the [issues](https://github.com/apache/dubbo/issues?q=is%3Aissue) and found no similar issues.

### Apache Dubbo Component

Java SDK (apache/dubbo)

### Dubbo Version

3.2.14

### Steps to reproduce this issue

my test code
client code

```
package org.example.client;

import org.apache.dubbo.config.ApplicationConfig;
import org.apache.dubbo.config.ConsumerConfig;
import org.apache.dubbo.config.MethodConfig;
import org.apache.dubbo.config.ReferenceConfig;
import org.apache.dubbo.config.bootstrap.DubboBootstrap;
import org.apache.dubbo.config.bootstrap.builders.ReferenceBuilder;
import org.apache.dubbo.remoting.Constants;
import org.apache.dubbo.rpc.FutureContext;
import org.example.api.GreetingsService;

import java.io.IOException;
import java.util.concurrent.*;

public class Application {

private static final ExecutorService worker = Executors.newFixedThreadPool(1, (r) ->{
Thread thread = new Thread(r);
thread.setName("worker-" + thread.getId());
return thread;
} );

private static final ExecutorService printer = Executors.newFixedThreadPool(1, (r) ->{
Thread thread = new Thread(r);
thread.setName("printer-"+thread.getId());
return thread;
});

public static void main(String[] args) throws IOException, InterruptedException {

ApplicationConfig application = new ApplicationConfig();
application.setName("client");

int requests = args.length > 0 ? Integer.parseInt(args[0]) : 100000;
String ip = args.length > 1 ? args[1] : "localhost";

MethodConfig methodConfig = new MethodConfig();
methodConfig.setName("sayHi");
methodConfig.setAsync(true);
methodConfig.setTimeout(1000000);

MethodConfig resetMethodConfig = new MethodConfig();
resetMethodConfig.setName("reset");
resetMethodConfig.setAsync(false);

ConsumerConfig consumerConfig = new ConsumerConfig();
consumerConfig.setCorethreads(1);
consumerConfig.setQueues(10000000);
consumerConfig.setTimeout(1000000);
consumerConfig.setThreads(1);

ReferenceConfig reference =
ReferenceBuilder.newBuilder()
.interfaceClass(GreetingsService.class)
.url("tri://"+ip+":50052")
.addMethod( methodConfig )
.addMethod( resetMethodConfig )
.consumer(consumerConfig)
.appendParameter(Constants.CONNECT_TIMEOUT_KEY, "1000000")
.build();
DubboBootstrap.getInstance()
.application(application)
.reference(reference)
.start();
GreetingsService service = reference.get();

service.reset(); //同步调用,重置服务端的计数器为0

CountDownLatch latch = new CountDownLatch(requests); // 创建一个计数器,用来记录请求的次数

int[] count = new int[1];
long start = System.currentTimeMillis();

for (int i = 0; i < requests; i++) {

service.sayHi("dubbo"); //这里的返回值为空,请不要使用
CompletableFuture future = FutureContext.getContext().getCompletableFuture();
worker.submit(() -> {
try {
String s = future.get();
latch.countDown();

printer.submit(() -> {
// if( ++count[0] % 1000 == 0 ) {
System.out.println("Receive result ======> " + s);
// }
});
} catch (InterruptedException e) {
throw new RuntimeException(e);
} catch (ExecutionException e) {
throw new RuntimeException(e);
}
});
}

try {
latch.await();
}catch (InterruptedException e)
{
e.printStackTrace();
}

long end = System.currentTimeMillis();
long delta = end - start;

Thread.sleep(1000);

System.out.println("总计" + requests + " 次");
System.out.println("耗时" + delta + " 毫秒");
System.out.println("TPS: " + requests * 1000L /delta + " 次/秒");

System.exit(0);
}

}

```

server code

```
package org.example.provider;

import org.apache.dubbo.config.ApplicationConfig;
import org.apache.dubbo.config.bootstrap.DubboBootstrap;
import org.apache.dubbo.config.bootstrap.builders.ProtocolBuilder;
import org.apache.dubbo.config.bootstrap.builders.ServiceBuilder;
import org.example.api.GreetingsService;

public class Application {
public static void main(String[] args) {

ApplicationConfig application = new ApplicationConfig();
application.setName("provider");

DubboBootstrap.getInstance()
.application(application)
.protocol(ProtocolBuilder.newBuilder()
.dispatcher("all")
.accepts(1000000)
.queues(1000000)
.threadpool("fixed")
.threads(1)
.name("tri")
.port(50052)
.build())
.service(
ServiceBuilder.newBuilder()
.interfaceClass(GreetingsService.class)
.ref(new GreetingsServiceImpl())
.build())
.start()
.await();
}
}
```

api code
```

package org.example.api;

public interface GreetingsService {
String sayHi(String name);

String reset();
}
```

api impl code
```

package org.example.provider;

import org.example.api.GreetingsService;

public class GreetingsServiceImpl implements GreetingsService {

private int count = 0;
// private AtomicInteger count = new AtomicInteger(0);

@Override
public String sayHi(String name) {
return "hi, " + ++count + "次 " + name;
}

@Override
public String reset() {
count = 0;
return "reset success";
}

}
```

server 192.168.253.176 startup commad
`java -cp Dubbo3Test-1.0-SNAPSHOT.jar:./lib/* org.example.provider.Application -Djava.net.preferIPv4Stack=true -Dio.netty.leakDetectionLevel=advanced -Xmx2048m -Xms2048m`

client test startup command
`java -Djava.net.preferIPv4Stack=true -Ddubbo.application.qos-port=33333 -Ddubbo.application.qos-enable=false -Dio.netty.leakDetectionLevel=advanced -Xmx2048m -Xms2048m -cp Dubbo3Test-1.0-SNAPSHOT.jar:lib/* org.example.client.Application 100000 192.168.253.176`

The VM specifications for both the client and server are 2C4G

tcp settings
```
root@ubuntu01:/home/zwb# sysctl -p
net.ipv4.tcp_rmem = 4096 832256 13320192
net.core.rmem_default = 2129920
net.core.rmem_max = 13320192
net.ipv4.tcp_mem = 44379 208128 524288
net.core.wmem_max = 13320192
net.ipv4.tcp_wmem = 4096 832256 13320192
```

ERROR occurs on the client. Procedure
```
09:40:35.711 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker: -| [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions.
09:40:35.711 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker: -| [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions.
09:40:35.711 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker: -| [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions.
09:40:35.712 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker: -| [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions.
09:40:35.712 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker: -| [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions.
09:40:35.712 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker: -| [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions.
09:40:35.712 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker: -| [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions.
09:40:35.712 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker: -| [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions.
09:40:35.712 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker: -| [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions.
09:40:35.712 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker: -| [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions.
09:40:35.712 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker: -| [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions.
09:40:35.712 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker: -| [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions.
09:40:35.712 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker: -| [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions.
09:40:35.712 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker: -| [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions.
09:40:35.712 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker: -| [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions.

```

WARN occurs on the server. Procedure
```
09:40:23.588 |-INFO [NettyServerWorker-5-1] ing.transport.netty4.NettyChannelHandler:56 -| [DUBBO] The connection of /192.168.253.201:44932 -> /192.168.253.176:50052 is established., dubbo version: 3.2.14, current host: 192.168.253.193
09:40:32.918 |-INFO [NettyServerWorker-5-1] ting.transport.netty4.NettyServerHandler:93 -| [DUBBO] The connection of /192.168.253.201:44932 -> /192.168.253.176:50052 is disconnected., dubbo version: 3.2.14, current host: 192.168.253.193
09:40:32.919 |-WARN [NettyServerWorker-5-1] .dubbo.remoting.transport.AbstractServer: -| [DUBBO] All clients has disconnected from /192.168.253.176:50052. You can graceful shutdown now., dubbo version: 3.2.14, current host: 192.168.253.193, error code: 99-0. This may be caused by unknown error in remoting module, go to https://dubbo.apache.org/faq/99/0 to find instructions.
09:40:32.925 |-INFO [NettyServerWorker-5-1] ing.transport.netty4.NettyChannelHandler:72 -| [DUBBO] The connection of /192.168.253.201:44932 -> /192.168.253.176:50052 is disconnected., dubbo version: 3.2.14, current host: 192.168.253.193
09:40:34.988 |-INFO [NettyServerWorker-5-2] ing.transport.netty4.NettyChannelHandler:56 -| [DUBBO] The connection of /192.168.253.201:36978 -> /192.168.253.176:50052 is established., dubbo version: 3.2.14, current host: 192.168.253.193

```

### What you expected to happen

The client program doesn't end as expected and keeps getting stuck

### Anything else

_No response_

### Are you willing to submit a pull request to fix on your own?

- [X] Yes I am willing to submit a pull request on my own!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.