brpc支持GPU Direct RDMA
- Dominant language
- C++
- Stars
- 17.6k
- Forks
- 4.1k
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 69
Description
**Is your feature request related to a problem?**
**Describe the solution you'd like**
在GPU上训练时使用brpc的RDMA传输,期望使用GDR能力,直接将**RPC请求**写入到GPU的buffer中,再通过cudamemcp或者gdrcopy从GPU上拷贝出协议部分进行解析,以减少CPU和GPU之间的数据传输。
**Describe alternatives you've considered**
进行的尝试:
将block_pool.cpp中的BlockPool替换成了通过cuda分配的显存,但在使用IOBuf时,会在分配出来的内存上进行placement new,

而这导致CPU直接访问显存,产生core dumped
因此,有以下三种想法:
1.修改block_pool.cpp,新增一个ObjectPool用于分配Block的实体,将Block的data指向BlockPool -> 即 Block的实体部分 与 数据部分分离
在此基础上,修改IOBuf的create block接口,objmem_allocate分配一个block出来,blockmem_allocate分配一段内存用于传输RPC请求
2.增加RDMA Write/Read接口;原本brpc的Send/Recv用于传输控制数据(例如,对端GPU地址),再使用RDMA Write直接将训练数据写到对端GPU地址上
3.使用append_with_user_data,将attachment从显存上分配,发送时使用RDMA的scatter-gatther特性,使其request请求写入到主机内存,attachment写入到显存上。
但是此接口也是会在分配出来的内存上进行placement new,因此也存在core dumped的问题,也需要修改IOBuf的接口
**Additional context/screenshots**
想请教这三个中是否有具有可行性的方案
Contributor guide
Research direction
Start by reading block_pool.cpp and the IOBuf block-allocation and append_with_user_data paths mentioned in the issue, then trace how placement new and RDMA Send/Recv currently handle buffers. The issue needs a concrete, agreed design among the three proposed approaches before implementation can be considered done.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- distributed-systems, networking
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100