dmlc / dmlc/mxnet-memonger

How to use on DDP

Open
#14 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
306
Forks
58
PR merge metrics
No merged PRs in 30d

Description

I used it on DDP ,and an error came out, I dont know if it can be used on DDP.
Error maybe occurred in loss.backward(). Thank you !

RuntimeError: Expected to mark a variable ready only once.
This error is caused by use of a module parameter outside the `forward` function.
The return value of the `forward` function is inspected by the distributed data parallel
wrapper to figure out if any of the module's parameters went unused.
If this is the case, it knows they won't receive gradients in a backward pass.
If any of those parameters are then used outside `forward`, this error condition is triggered.
You can disable unused parameter detection by passing the keyword argument
`find_unused_parameters=False` to `torch.nn.parallel.DistributedDataParallel`.

Contributor guide

No contributing guide indexed for this repository

Research direction

No source file or test is named. Start by reproducing the reported DistributedDataParallel setup and inspect the failure at loss.backward(), then trace how the module is wrapped and how parameters are used around forward. Done means establishing whether DDP is supported and documenting the required usage or identifying the incompatibility.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.