[Bug] RuntimeError: [3] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: Socket Timeout
- Dominant language
- Python
- Stars
- 421
- Forks
- 67
- PR merge metrics
- No merged PRs in 30d
Description
### Describe the bug
It's a probabilistic occurrence, Socket Timeout when group.allreduce([tensor], opts)
if group in _world.pg_coalesce_state.keys():
# We are in coalescing context, do not issue single operation, just append a collective representation
coll = _CollOp(all_reduce, tensor, None, op, None)
_world.pg_coalesce_state[group].append(coll)
_world.pg_coalesce_state[group].append(coll)
_world.pg_coalesce_state[group].append(coll)
_world.pg_coalesce_state[group].append(coll)
if async_op:
return _IllegalWork()
else:
return None
> work = group.allreduce([tensor], opts)
E RuntimeError: [3] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: Socket Timeout
E Exception raised from doWait at ../torch/csrc/distributed/c10d/TCPStore.cpp:445 (most recent call first):
More information: https://github.com/InternLM/InternEvo/actions/runs/10012995770/job/27889229378
### Environment
python3.10
torch2.1
### Other information
_No response_
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.