How to use on DDP
- Dominant language
- Python
- Stars
- 306
- Forks
- 58
- PR merge metrics
- No merged PRs in 30d
Description
I used it on DDP ,and an error came out, I dont know if it can be used on DDP.
Error maybe occurred in loss.backward(). Thank you !
RuntimeError: Expected to mark a variable ready only once.
This error is caused by use of a module parameter outside the `forward` function.
The return value of the `forward` function is inspected by the distributed data parallel
wrapper to figure out if any of the module's parameters went unused.
If this is the case, it knows they won't receive gradients in a backward pass.
If any of those parameters are then used outside `forward`, this error condition is triggered.
You can disable unused parameter detection by passing the keyword argument
`find_unused_parameters=False` to `torch.nn.parallel.DistributedDataParallel`.
Contributor guide
No contributing guide indexed for this repository
Research direction
No source file or test is named. Start by reproducing the reported DistributedDataParallel setup and inspect the failure at loss.backward(), then trace how the module is wrapped and how parameters are used around forward. Done means establishing whether DDP is supported and documenting the required usage or identifying the incompatibility.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100