[mcore] Dynamic CP group for training with sequence packing
- Dominant language
- Python
- Stars
- 2k
- Forks
- 561
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 145
Description
**Is your feature request related to a problem? Please describe.**
In long-context GRPO, the sequence length distribution in a batch can be very skewed, and always using a large CP to tolerate the longest sequence is inefficient for most short sequences.
This issue requests for dynamic CP group feature that can dynamically arrange DP and CP group size per batch to reduce exposed CP communication.
**Describe the solution you'd like**
A clear and concise description of what you want to happen.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Add any other context or screenshots about the feature request here.
Contributor guide
Assessment
This issue has not been assessed yet.