[ENHANCEMENT] Context Parallel Support for Mamba2
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 4.5k
- Avg merge
- 4d 6h
- Merged PRs (30d)
- 271
Description
**Is your feature request related to a problem? Please describe.**
Megatron LM currently supports context parallelism (CP) for GPT-based models, but it does not look like CP is supported for Mamba2 models.
**Describe the solution you'd like**
It would be great to extend CP support to Mamba2 SSM layers.
**Describe alternatives you've considered**
We have used other forms of parallelism, such as tensor, pipeline, and sequence to train Mamba-based models, but context parallelism is lacking and could be very useful for certain applications such as DNA sequence modeling.
**Proposed implementation**
A possible solution for extending CP support to Mamba has been proposed in the mamba_ssm repository: https://github.com/state-spaces/mamba/pull/664
Contributor guide
Assessment
This issue has not been assessed yet.