NVIDIA / NVIDIA/Megatron-LM

[Design Doc] HybridModel - Next Gen Model Class

Open
#4,620 1 comment 4 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

The new `HybridModel` class is a “heterogenous layer” model built to replace the existing [GPTModel](https://github.com/NVIDIA/Megatron-LM/blob/3b1521e29c96b4613a70545cfea0d2c813a85ad0/megatron/core/models/gpt/gpt_model.py#L43) and `MambaModel` classes. The `HybridModel` class will be used as the foundation for future Nemotron models and next-generation open-source models. It will be highly-customizable to support varying layer types and sizes.

Please feel free to read our [design doc](https://github.com/user-attachments/files/30916074/HybridModel.pdf) and ask any follow-up questions in this issue!

Authors: @Phlip79, @janEbert

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.