intel / intel/torch-xpu-ops

[distributed] RuntimeError: No backend for the parent process group or its backend does not support splitting in test/distributed/test_device_mesh.py

Open
#3,233 3 comments 0 reactions 3 assignees Claimed by @EikanWang View on GitHub
bug module: distributed
Dominant language
Python
Stars
113
Forks
128
Avg merge
5d 13h
Merged PRs (30d)
107

Description

### 🐛 Describe the bug

please get whls from https://github.com/intel/torch-xpu-ops/actions/runs/23637238144 or use or use gh download
```
gh run download 23637238144 --repo intel/torch-xpu-ops --name Torch-XPU-Wheel-1826-23637238144-1 --dir path --pattern "*.zip"
git clone -b distributed_2.11 https://github.com/daisyden/pytorch.git
cd pytorch
pip install -r requirements.txt
pip install pytest expecttest
pytest -v test/distributed/test_device_mesh.py::DeviceMeshTest::test_2d_mesh_eager_init_subgroup
pytest -v test/distributed/test_device_mesh.py::InitDeviceMeshTest::test_backend_override_argument_dict_with_idx_and_backend_eager
```
```
Traceback (most recent call last):
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 986, in run_test
getattr(self, test_name)()
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 833, in wrapper
fn()
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3365, in wrapper
method(*args, **kwargs)
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/testing/_internal/distributed/_tensor/common_dtensor.py", line 902, in wrapper
raise e
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/testing/_internal/distributed/_tensor/common_dtensor.py", line 899, in wrapper
func(self, *args, **kwargs) # type: ignore[misc]
File "/home/sdp/xiangdong/pytorch/test/distributed/test_device_mesh.py", line 205, in test_2d_mesh_eager_init_subgroup
mesh_2d = init_device_mesh(self.device_type, mesh_shape)
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/distributed/device_mesh.py", line 1588, in init_device_mesh
device_mesh = DeviceMesh(
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/distributed/device_mesh.py", line 310, in __init__
self._dim_group_names = self._init_process_groups(
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/distributed/device_mesh.py", line 611, in _init_process_groups
DeviceMesh._init_one_process_group(
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/distributed/device_mesh.py", line 561, in _init_one_process_group
dim_group = split_group(
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 97, in wrapper
func_return = func(*args, **kwargs)
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 5485, in split_group
raise RuntimeError(
RuntimeError: No backend for the parent process group or its backend does not support splitting
```

### Versions

https://github.com/daisyden/pytorch/tree/distributed_2.11

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.