deepspeedai / deepspeedai/DeepSpeed
some questions about zero-1 stage the source code
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
Reduce Scatter for Stage 1
After the gradient communication is completed, the updated grads is saved in single_comm_all_partitions, which is a local variable, but in the step function, the updated grad value is obtained from local_grad_sub_partitions, and its value is obtained from params_in_rank_sub_partitions，to my understanding , Params_in_rank_sub_partitions has not updated grads.
is there some shallow copy mechanism here?
I am really looking forward to reply
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the Reduce Scatter for Stage 1 flow through the step function and the symbols single_comm_all_partitions, local_grad_sub_partitions, and params_in_rank_sub_partitions. Check how these values are assigned and whether shared or shallow state is involved. Done means clearly explaining whether the updated gradients reach the value used by step.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 18/100