NVIDIA / NVIDIA/Megatron-LM

[ENHANCEMENT] Context Parallel Support for Mamba2

Open
#1,437 5 comments 3 reactions 0 assignees View on GitHub
community-request enhancement waiting-on-customer
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

**Is your feature request related to a problem? Please describe.**
Megatron LM currently supports context parallelism (CP) for GPT-based models, but it does not look like CP is supported for Mamba2 models.

**Describe the solution you'd like**
It would be great to extend CP support to Mamba2 SSM layers.

**Describe alternatives you've considered**
We have used other forms of parallelism, such as tensor, pipeline, and sequence to train Mamba-based models, but context parallelism is lacking and could be very useful for certain applications such as DNA sequence modeling.

**Proposed implementation**
A possible solution for extending CP support to Mamba has been proposed in the mamba_ssm repository: https://github.com/state-spaces/mamba/pull/664

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.