AI-Hypercomputer / AI-Hypercomputer/gpu-recipes
Network gIB Issues
- 主要语言
- Python
- 星标
- 140
- 派生
- 87
- 平均合并
- 7 天 21 小时
- 30 天内合并 PR
- 1
描述
I'm trying to run any of these tests and can't. As far as I can tell, I have everything configured according to the prerequisites and documentation. Everything seems to work until the containers spin up and run. The best I can tell, the actual error is this:
```
torch.distributed.DistBackendError: NCCL error in: /opt/pytorch/pytorch/torch/csrc/distributed/c10d/NCCLUtils.hpp:328, invalid usage (run with NCCL_DEBUG=WARN for details), NCCL version 2.25.1
ncclInvalidUsage: This usually reflects invalid usage of NCCL library.
Last error:
Error: network gIB not found.
```
Does anyone have any advice? I'm trying to do this with FlexStart node pools and I'm not sure if there is a network configuration difference between DENSE provisioned nodes (mentioned in the readme) and FlexStart. Any help is appreciated.
贡献指南
评估
这个 Issue 还没有评估数据。