example/rdma_performance/client.cpp:155] RPC call failed: [E1008]Reached timeout=2000ms @22.22.30.6:8004
- Dominant language
- C++
- Stars
- 17.6k
- Forks
- 4.1k
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 69
Description
**Describe the bug (描述bug)**
When running rdma_performance built from rdma-old, it reports below errro:
```
example/rdma_performance/client.cpp:155] RPC call failed: [E1008]Reached timeout=2000ms @22.22.30.6:8004
```
**To Reproduce (复现方法)**
1. Build rdma-old branch
```
$ git clone https://github.com/apache/incubator-brpc.git verbs_old
$ cd verbs_old
$ git checkout rdma-old
$ bash config_brpc.sh --headers=/usr/include --libs=/usr/lib64
$ mkdir build
$ cd build
$ cmake -DWITH_RDMA=on ..
$ make
```
2. Build rdma-performance
```
$ cd verbs_old/example/rdma_performance
$ mkdir build
$ cd build
$ cmake ..
$ make
```
3. Start running two server process on one host, suppose the bond1 ip address is 22.22.30.6
```
verbs_old/example/rdma_performance/build/]$ numactl -i netdev:ens3f0np0 ./perf_server --use_rdma=true --rdma_device=mlx5_bond_1 --port=8003
```
&
```
verbs_old/example/rdma_performance/build/]$ numactl -i netdev:ens3f0np0 ./perf_server --use_rdma=true --rdma_device=mlx5_bond_1 --port=8004
```
4. Start running one client process on one host, suppose the bond1 ip address is 22.22.30.7
```
verbs_old/example/rdma_performance/build/]$ numactl -i netdev:ens3f0np0 ./perf_client --use_rdma=true --rdma_device=mlx5_bond_1 --servers=22.22.30.6:8003+22.22.30.6:8004 --echo_attachment=false --attachment_size=16384 --thread_num=16
```
Sometimes, the client process will reports below error:
```
verbs_old/example/rdma_performance/client.cpp:155] RPC call failed: [E1008]Reached timeout=2000ms @22.22.30.6:8004
verbs_old/example/rdma_performance/client.cpp:155] RPC call failed: [E1008]Reached timeout=2000ms @22.22.30.6:8003
```
**Expected behavior (期望行为)**
There should be no Reached timeout error.
The client and server host are connected through one switch. the RTT is very small.
**Versions (各种版本)**
OS:
```
$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="8"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"
```
Compiler:
```
$ gcc --version
gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-4)
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
```
brpc:
```
rdma-old branch
```
protobuf:
```
$ rpm -qa | grep protobuf
protobuf-c-1.3.0-6.el8.x86_64
protobuf-devel-3.5.0-13.el8.x86_64
protobuf-compiler-3.5.0-13.el8.x86_64
protobuf-3.5.0-13.el8.x86_64
```
**Additional context/screenshots (更多上下文/截图)**
Contributor guide
Research direction
Start with example/rdma_performance/client.cpp at line 155 and reproduce the timeout using the rdma-old build, perf_server, and perf_client commands in the issue. Compare behavior with two servers on ports 8003 and 8004 and the specified RDMA settings. Done means the reproduction no longer reports E1008 timeout errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- networking, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100