deepspeedai / deepspeedai/DeepSpeed
Data Loading for DeepSpeed Ulysses and Data Parallelism
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
Describe the bug
After reading the source code here, here and some existing issues such as https://github.com/microsoft/DeepSpeed/issues/4359#issuecomment-1726166408, it seems like if we want to use Ulysses with Data Parallelism, we need to modify the data loader such that for each sequence, each GPU needs to load 1/P (assume we have P GPUs) of the sequence.
If my understanding is incorrect, it would be great if you could let me know what Ulysses actually does and, if needed, how we should modify the data loader to support it. The blog post and paper are not very clear about the implementation details.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading deepspeed/sequence/layer.py at the linked lines and trace how Ulysses interacts with data parallelism. Check the existing issue discussion referenced in the report; done means establishing whether the data loader needs changes and documenting the required behavior or implementation scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100