pytorch / pytorch/gloo

[BUG] Does ibverbs transport support transferring large volumes of data?

Open
#474 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
1.5k
Forks
368
Avg merge
19h 32m
Merged PRs (30d)
3

Description

It seems that ibverbs transport doesn't support transferring large volumes of data.

The max_msg_sz of my device is 1GB (0x40000000) and when I transfer data that larger than 1GB (like 1GB + 4bytes), an error occured as below.

$ ibv_devinfo -v | grep max_msg_sz
			max_msg_sz:		0x40000000
			max_msg_sz:		0x40000000
			max_msg_sz:		0x40000000
			max_msg_sz:		0x40000000
			max_msg_sz:		0x40000000
			max_msg_sz:		0x40000000
			max_msg_sz:		0x40000000
			max_msg_sz:		0x40000000

command

./gloo/benchmark/benchmark -s 2 -r 0 -h 110.110.8.158 -p 6379 -x 123 -t ibverbs --no-verify broadcast --messages 1 --elements 268435457 
./gloo/benchmark/benchmark -s 2 -r 1 -h 110.110.8.158 -p 6379 -x 123 -t ibverbs --no-verify broadcast --messages 1 --elements 268435457 

error

[/nvme1/chenxin/ws/test/gloo/gloo/transport/ibverbs/pair.cc:587] ERROR LID: 0 QPN: 5728 PSN: 3360883->LID: 0 QPN: 5727 PSN: 3360883: Exception in handleCompletion: [enforce fail at /nvme1/chenxin/ws/test/gloo/gloo/transport/ibverbs/pair.cc:681] wc->status == IBV_WC_SUCCESS. 1 vs 0. Memory region recv for slot 0: local length error
[/nvme1/chenxin/ws/test/gloo/gloo/transport/ibverbs/device.cc:230] ERROR Exception while handling completion event: [enforce fail at /nvme1/chenxin/ws/test/gloo/gloo/transport/ibverbs/pair.cc:681] wc->status == IBV_WC_SUCCESS. 1 vs 0. Memory region recv for slot 0: local length error
terminate called after throwing an instance of 'gloo::EnforceNotMet'
  what():  [enforce fail at /nvme1/chenxin/ws/test/gloo/gloo/transport/ibverbs/pair.cc:681] wc->status == IBV_WC_SUCCESS. 1 vs 0. Memory region recv for slot 0: local length error
Aborted (core dumped)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in gloo/transport/ibverbs/pair.cc around the completion error at lines 587 and 681, then inspect gloo/transport/ibverbs/device.cc around line 230. Reproduce the failure with the supplied benchmark commands using a message larger than 1GB, and determine whether the ibverbs transport can complete it without the local length error.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.