apache / apache/brpc

rdma的IOBuf::append_user_data_with_meta无法传输4G以上的数据

Open
#2,475 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
17.6k
Forks
4.1k
Avg merge
2d 12h
Merged PRs (30d)
69

Description

**Describe the bug (描述bug)**
自己写了一个[测试](https://github.com/binarycopycode/brpc_rdma_tcp_perf)rdma和tcp速度差别的小测试,自己测试了128,256,512Mb都没问题。当传输大小为1G的数据时,rdma无任何报错,但是client接收到的attachment().size()是0,而使用tcp传输大小为1G的数据时,则出现报错

```
W1220 15:39:25.600674 5980 /home/binarycopycode/work/brpc_rdma_tcp_perf/src/brpc/input_messenger.cpp:375] Fail to read from Socket{id=102 fd=9 addr=192.168.182.129:40614:8011} (0x7f0d58026820): Connection reset by peer
W1220 15:39:25.600733 5980 /home/binarycopycode/work/brpc_rdma_tcp_perf/src/brpc/socket.cpp:1774] Fail to keep-write into Socket{id=102 fd=9 addr=192.168.182.129:40614:8011} (0x7f0d58026820): Broken pipe
```

**To Reproduce (复现方法)**
按照https://github.com/binarycopycode/brpc_rdma_tcp_perf 编译
测试rdma,注意我的池子大小和brpc的max_body_size都设为了2G,注意修改运行client命令的ip地址
`./server -port=8010 -data_size_mb=1023 -rdma_memory_pool_initial_size_mb=2048 -max_body_size=2147483648`
`./client -server=192.168.182.129:8010 -rdma_memory_pool_initial_size_mb=2048 -max_body_size=2147483648`
测试tcp,加上-use_rdma=false即可
`./server -use_rdma=false -port=8011 -data_size_mb=1023 -rdma_memory_pool_initial_size_mb=2048 -max_body_size=2147483648`

**Expected behavior (期望行为)**

client能正常接收attachment

**Versions (各种版本)**
OS: CentOS8.5
Compiler: g++ (GCC) 8.5.0 20210514 (Red Hat 8.5.0-4)
brpc:
protobuf: libprotoc 3.5.0

**Additional context/screenshots (更多上下文/截图)**
![rdma error](https://github.com/apache/brpc/assets/34117975/13be872e-8086-4383-9c77-9c67b7f7c011)
![tcp error](https://github.com/apache/brpc/assets/34117975/3662bcac-7bb2-4eaa-85db-8a25a4faddce)

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the RDMA and TCP commands from the linked brpc_rdma_tcp_perf test, then inspect IOBuf::append_user_data_with_meta and the read/write paths named in src/brpc/input_messenger.cpp and src/brpc/socket.cpp. Compare behavior around the reported large payload sizes; done means the client receives the expected attachment without the reported reset, broken-pipe, or empty-result behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.