sgl-project / sgl-project/SpecForge
[RFC]: Integrate USP (Ulysses + Ring Attention) for Context Parallelism in SpecForge
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 347
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 41
Description
1. Motivation
Training 16k-length sequences currently causes OOM errors https://github.com/sgl-project/SpecForge/issues/112. To support 100k+ sequences, we need efficient context parallelism (CP). Per https://arxiv.org/abs/2405.07719, USP (Ulysses + Ring Attention) outperforms standalone approaches, making it our top choice.
2. Proposal
Integrate USP into SpecForge. This hybrid approach combines:
Ulysses: Offers better performance
Ring Attention: Enables support for longer sequence lengths
3. Expected Benefits
Enable 100k+ sequence training without OOM
Maintain computational efficiency
Preserve model accuracy at scale
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading issue #112 to understand the existing 16k-sequence OOM failure, then review the linked USP paper. Map how SpecForge currently handles context parallelism and determine the integration scope. Done means supporting 100k+ sequence training without OOM while preserving computational efficiency and model accuracy.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100