AI-Hypercomputer / AI-Hypercomputer/gpu-recipes
Network gIB Issues
- 主要言語
- Python
- スター
- 140
- フォーク
- 87
- 平均マージ
- 7日 21時間
- マージ済み PR（30日）
- 1
説明
I'm trying to run any of these tests and can't. As far as I can tell, I have everything configured according to the prerequisites and documentation. Everything seems to work until the containers spin up and run. The best I can tell, the actual error is this:
```
torch.distributed.DistBackendError: NCCL error in: /opt/pytorch/pytorch/torch/csrc/distributed/c10d/NCCLUtils.hpp:328, invalid usage (run with NCCL_DEBUG=WARN for details), NCCL version 2.25.1
ncclInvalidUsage: This usually reflects invalid usage of NCCL library.
Last error:
Error: network gIB not found.
```
Does anyone have any advice? I'm trying to do this with FlexStart node pools and I'm not sure if there is a network configuration difference between DENSE provisioned nodes (mentioned in the readme) and FlexStart. Any help is appreciated.
コントリビューションガイド
評価
この issue はまだ評価されていません。