[Design Doc] HybridModel - Next Gen Model Class
Open
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 4.5k
- Avg merge
- 4d 6h
- Merged PRs (30d)
- 271
Description
The new `HybridModel` class is a “heterogenous layer” model built to replace the existing [GPTModel](https://github.com/NVIDIA/Megatron-LM/blob/3b1521e29c96b4613a70545cfea0d2c813a85ad0/megatron/core/models/gpt/gpt_model.py#L43) and `MambaModel` classes. The `HybridModel` class will be used as the foundation for future Nemotron models and next-generation open-source models. It will be highly-customizable to support varying layer types and sizes.
Please feel free to read our [design doc](https://github.com/user-attachments/files/30916074/HybridModel.pdf) and ask any follow-up questions in this issue!
Authors: @Phlip79, @janEbert
Contributor guide
Assessment
This issue has not been assessed yet.