[BUG] Context parallelism not enabled in SFT
- Dominant language
- Python
- Stars
- 450
- Forks
- 40
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the bug**
get_batch doesnt support cp broadcasting. Running with cp > 1 on current code yields wrong loss
**To Reproduce**
any same SFT experiment with cp=1 then cp=2
**Expected behavior**
cp=1 or cp=2 should yield very similar lm loss
**Screenshots**
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the SFT get_batch implementation and the context-parallel broadcasting path. Reproduce the same SFT experiment with cp=1 and cp=2, then compare the language-model loss. Done means context parallelism is enabled for batching and the losses are very similar across both settings.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100