deepspeedai / deepspeedai/DeepSpeed
detect redundant tensors in param group
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
When creating the optimizer, check for parameter duplicates in the param_group. Currently, we will create memory for all tensors in the param_groups even if they internally point to the same tensor data.
This can be an issue with models containing tied parameters. Since these parameters could be member variable of different modules, they will be detected as different parameters by torch, even though they maybe pointing to the same tensor.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating optimizer creation and parameter-group handling. Reproduce the case with tied parameters referenced by different modules, then verify that internally duplicated tensors do not receive duplicate optimizer memory. Done means redundant tensor data is detected and handled without changing behavior for distinct parameters.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100