intel / intel/torch-xpu-ops

[distributed] /handler/dump_nccl_trace_pickle and nccl_log need in distributed ut tests

Open
#3,100 0 comments 0 reactions 1 assignee Claimed by @songhappy View on GitHub
module: distributed
Dominant language
Python
Stars
113
Forks
128
Avg merge
5d 13h
Merged PRs (30d)
107

Description

### 🐛 Describe the bug

```
git clone -b distributed_2.11 https://github.com/daisyden/pytorch.git
cd pytorch
pip install -r requirements.txt
pip install pytest expecttest
pytest -v test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardForceSumReduction::test_fully_shard_force_sum_both_reductions
pytest -v test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardForceSumReduction::test_fully_shard_force_sum_reduce_scatter
pytest -v test/distributed/elastic/test_control_plane.py::WorkerServerTest::test_dump_nccl_trace_pickle
pytest -v test/distributed/elastic/test_control_plane.py::WorkerServerTest::test_dump_nccl_trace_pickle_with_json
pytest -v test/distributed/elastic/test_control_plane.py::WorkerServerTest::test_dump_nccl_trace_pickle_with_params
```
```
Traceback (most recent call last):
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 986, in run_test
getattr(self, test_name)()
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 833, in wrapper
fn()
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3364, in wrapper
method(*args, **kwargs)
File "/home/sdp/miniforge3/envs/xccl_test/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 241, in wrapper
return func(*args, **kwargs)
File "/home/sdp/xiangdong/pytorch/test/distributed/_composable/fsdp/test_fully_shard_comm.py", line 1755, in test_fully_shard_force_sum_reduce_scatter
with open(self.nccl_log_dir.name + "/nccl_log") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmp2cavipze/nccl_log'
```

### Versions

https://github.com/daisyden/pytorch/tree/distributed_2.11

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.