InternLM / InternLM/InternEvo

[Bug] RuntimeError: [3] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: Socket Timeout

Open
#285 0 comments 0 reactions 1 assignee Claimed by @sunpengsdu View on GitHub
bug
Dominant language
Python
Stars
421
Forks
67
PR merge metrics
No merged PRs in 30d

Description

### Describe the bug

It's a probabilistic occurrence, Socket Timeout when group.allreduce([tensor], opts)
if group in _world.pg_coalesce_state.keys():
# We are in coalescing context, do not issue single operation, just append a collective representation
coll = _CollOp(all_reduce, tensor, None, op, None)
_world.pg_coalesce_state[group].append(coll)
_world.pg_coalesce_state[group].append(coll)
_world.pg_coalesce_state[group].append(coll)
_world.pg_coalesce_state[group].append(coll)
if async_op:
return _IllegalWork()
else:
return None

> work = group.allreduce([tensor], opts)
E RuntimeError: [3] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: Socket Timeout
E Exception raised from doWait at ../torch/csrc/distributed/c10d/TCPStore.cpp:445 (most recent call first):
More information: https://github.com/InternLM/InternEvo/actions/runs/10012995770/job/27889229378

### Environment

python3.10
torch2.1

### Other information

_No response_

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.