KeyError when using DDP
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 9k
- Forks
- 1.5k
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 3
Description
Using DistributedDataParallel results a KeyError with the following trace
Traceback (most recent call last):
File "training_script.py", line 328, in iteration
loss.backward()
File "/anaconda3/lib/python3.7/site-packages/torch/tensor.py", line 102, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/anaconda3/lib/python3.7/site-packages/torch/autograd/__init__.py", line 90, in backward
allow_unreachable=True) # allow_unreachable flag
File "/anaconda3/lib/python3.7/site-packages/apex-0.1-py3.7-linux-x86_64.egg/apex/parallel/distributed.py", line 362, in allreduce_hook
self.comm_ready_buckets(param)
File "/anaconda3/lib/python3.7/site-packages/apex-0.1-py3.7-linux-x86_64.egg/apex/parallel/distributed.py", line 422, in comm_ready_buckets
bucket_idx, bucket_loc = self.param_id_to_bucket[id(param)]
KeyError: 139779067778824
pytorch: 1.0.0
apex compiled without --cpp_ext
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in apex/parallel/distributed.py, especially allreduce_hook and comm_ready_buckets, where the traceback shows the KeyError at param_id_to_bucket lookup. Reproduce the reported DistributedDataParallel setup with PyTorch 1.0.0 and Apex built without --cpp_ext; done means the backward pass no longer raises this KeyError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100