Refactor using CP inside loss function
Open
enhancement
t-pytdensor
- Dominant language
- Python
- Stars
- 2k
- Forks
- 561
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 145
Description
Some nits for CP on dtensor. This could be confusing for developers. But it has to be the state right now. Just file an issue to track this.
1. Mcore explicitly use context parallel group.
2. Dtensor relies on the seq_index.
We will eventually switch to flex attention when it is ready for cp.
Contributor guide
Assessment
This issue has not been assessed yet.