[BUG] Does ibverbs transport support transferring large volumes of data?
Open
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 1.5k
- Forks
- 368
- Avg merge
- 19h 32m
- Merged PRs (30d)
- 3
Description
It seems that ibverbs transport doesn't support transferring large volumes of data.
The max_msg_sz of my device is 1GB (0x40000000) and when I transfer data that larger than 1GB (like 1GB + 4bytes), an error occured as below.
$ ibv_devinfo -v | grep max_msg_sz
max_msg_sz: 0x40000000
max_msg_sz: 0x40000000
max_msg_sz: 0x40000000
max_msg_sz: 0x40000000
max_msg_sz: 0x40000000
max_msg_sz: 0x40000000
max_msg_sz: 0x40000000
max_msg_sz: 0x40000000
command
./gloo/benchmark/benchmark -s 2 -r 0 -h 110.110.8.158 -p 6379 -x 123 -t ibverbs --no-verify broadcast --messages 1 --elements 268435457
./gloo/benchmark/benchmark -s 2 -r 1 -h 110.110.8.158 -p 6379 -x 123 -t ibverbs --no-verify broadcast --messages 1 --elements 268435457
error
[/nvme1/chenxin/ws/test/gloo/gloo/transport/ibverbs/pair.cc:587] ERROR LID: 0 QPN: 5728 PSN: 3360883->LID: 0 QPN: 5727 PSN: 3360883: Exception in handleCompletion: [enforce fail at /nvme1/chenxin/ws/test/gloo/gloo/transport/ibverbs/pair.cc:681] wc->status == IBV_WC_SUCCESS. 1 vs 0. Memory region recv for slot 0: local length error
[/nvme1/chenxin/ws/test/gloo/gloo/transport/ibverbs/device.cc:230] ERROR Exception while handling completion event: [enforce fail at /nvme1/chenxin/ws/test/gloo/gloo/transport/ibverbs/pair.cc:681] wc->status == IBV_WC_SUCCESS. 1 vs 0. Memory region recv for slot 0: local length error
terminate called after throwing an instance of 'gloo::EnforceNotMet'
what(): [enforce fail at /nvme1/chenxin/ws/test/gloo/gloo/transport/ibverbs/pair.cc:681] wc->status == IBV_WC_SUCCESS. 1 vs 0. Memory region recv for slot 0: local length error
Aborted (core dumped)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in gloo/transport/ibverbs/pair.cc around the completion error at lines 587 and 681, then inspect gloo/transport/ibverbs/device.cc around line 230. Reproduce the failure with the supplied benchmark commands using a message larger than 1GB, and determine whether the ibverbs transport can complete it without the local length error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- distributed-systems, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100