deepspeedai / deepspeedai/DeepSpeed
Questions about the AWM in the ZeRO-Infinity paper
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
Hello, I'm reading the ZeRO-Infinity paper. I have some questions about Equation (5) (In Section 3 MEMORY REQUIREMENTS).
- This estimation is the size of the total activation per Transformer block (ci = 1). Is it still correct for ci > 1? Looks like the activation between blocks is not considered in this equation.
- Could you please explain the second term (2 x attn_heads x seq) in detail? I'm a little confused by it.
Thanks a lot.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read Section 3, “Memory Requirements,” of the ZeRO-Infinity paper and inspect Equation (5). A complete response should address whether the estimate remains valid for ci > 1 and explain the 2 × attn_heads × seq term, with the role of activations between blocks made explicit.
Written by the indexing model from the issue text.
Assessment
- Domain
- distributed-systems, machine-learning
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100