Big memory usage different between torch.nn.parallel.DDP and apex.parallel.DDP
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 9k
- Forks
- 1.5k
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 3
Description
When I use torch.nn.parallel.DDP with O1, the memory usage is about 9800MB on my RTX2080Ti.
However, when I replace torch.nn.parallel.DDP with apex.parallel.DDP without any other changes, the memory usage is about 5000MB, which is far much less.
Is this situation normal?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The report compares torch.nn.parallel.DDP and apex.parallel.DDP with O1 on an RTX2080Ti, but names no repository files or tests. Begin by reproducing the two configurations and tracing their memory allocation behavior; the work is complete when the difference is explained and any confirmed issue has a clear resolution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100