[distributed] RuntimeError: Work ran time out after 0 milliseconds with test_distributed_spawn.py
- Dominant language
- Python
- Stars
- 113
- Forks
- 128
- Avg merge
- 5d 13h
- Merged PRs (30d)
- 107
Description
### 🐛 Describe the bug
git clone -b madhumitha0102/distributed_2.10 https://github.com/madhumitha0102/pytorch
##Create pytorch wheel using this repo
#source oneapi variables and env variables necessary for the build
python setup.py clean
make triton
MAX_JOBS=32 python setup.py bdist_wheel
cd pytorch
pip install pytest expecttest
pip install -r requirements.txt
``` touch /tmp/barrier && TEMP_DIR=/tmp BACKEND=xccl WORLD_SIZE=4 pytest test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_with_logger ```
``` touch /tmp/barrier && TEMP_DIR=/tmp BACKEND=xccl WORLD_SIZE=4 pytest test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks ```
``` Traceback (most recent call last):
File "/mnt/scratch/pytorch_distributed_workspace/miniforge3/envs/pytorch_develop_env/lib/python3.10/unittest/case.py", line 59, in testPartExecutor
yield
File "/mnt/scratch/pytorch_distributed_workspace/miniforge3/envs/pytorch_develop_env/lib/python3.10/unittest/case.py", line 591, in run
self._callTestMethod(testMethod)
File "/mnt/scratch/pytorch_distributed_workspace/miniforge3/envs/pytorch_develop_env/lib/python3.10/unittest/case.py", line 549, in _callTestMethod
method()
File "/mnt/scratch/pytorch_distributed_workspace/uts_misc/pytorch/torch/testing/_internal/common_distributed.py", line 814, in wrapper
self._join_processes(fn)
File "/mnt/scratch/pytorch_distributed_workspace/uts_misc/pytorch/torch/testing/_internal/common_distributed.py", line 1083, in _join_processes
self._check_return_codes(fn, elapsed_time)
File "/mnt/scratch/pytorch_distributed_workspace/uts_misc/pytorch/torch/testing/_internal/common_distributed.py", line 1123, in _check_return_codes
raise RuntimeError(error)
RuntimeError: Process 0 exited with error code 10 and exception:
Traceback (most recent call last):
File "/mnt/scratch/pytorch_distributed_workspace/uts_misc/pytorch/torch/testing/_internal/common_distributed.py", line 969, in run_test
getattr(self, test_name)()
File "/mnt/scratch/pytorch_distributed_workspace/uts_misc/pytorch/torch/testing/_internal/common_distributed.py", line 816, in wrapper
fn()
File "/mnt/scratch/pytorch_distributed_workspace/uts_misc/pytorch/torch/testing/_internal/common_utils.py", line 3331, in wrapper
method(*args, **kwargs)
File "/mnt/scratch/pytorch_distributed_workspace/uts_misc/pytorch/torch/testing/_internal/common_distributed.py", line 364, in wrapper
ret = func(*args, **kwargs)
File "/mnt/scratch/pytorch_distributed_workspace/uts_misc/pytorch/torch/testing/_internal/common_distributed.py", line 233, in wrapper
return func(*args, **kwargs)
File "/mnt/scratch/pytorch_distributed_workspace/uts_misc/pytorch/torch/testing/_internal/distributed/distributed_test.py", line 8898, in test_monitored_barrier_allreduce_hang
self._test_monitored_barrier_allreduce_hang(wait_all_ranks=False)
File "/mnt/scratch/pytorch_distributed_workspace/uts_misc/pytorch/torch/testing/_internal/distributed/distributed_test.py", line 8890, in _test_monitored_barrier_allreduce_hang
self._barrier(timeout=30)
File "/mnt/scratch/pytorch_distributed_workspace/uts_misc/pytorch/torch/testing/_internal/distributed/distributed_test.py", line 654, in _barrier
Barrier.sync(*args, **kwargs)
File "/mnt/scratch/pytorch_distributed_workspace/uts_misc/pytorch/torch/testing/_internal/distributed/distributed_test.py", line 571, in sync
raise RuntimeError("barrier timeout")
RuntimeError: barrier timeout
```
### Versions
PyTorch: https://github.com/madhumitha0102/pytorch
Contributor guide
Assessment
This issue has not been assessed yet.