[Deprecation] `GPTModel`
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 4.5k
- Avg merge
- 4d 3h
- Merged PRs (30d)
- 272
Description
We are deprecating `GPTModel` in favor of `HybridModel`. **We will only be accepting critical bug fixes for `GPTModel`.**
## FAQs
**Why `HybridModel`?**
1. HybridModel is purpose-built for heterogeneity
2. It supports arbitrary layer placements, SSM layers, and flexible VPP
3. Architecture is more user-friendly, customizable, and intuitive
4. Large reduction in complexity relative to GPTModel
For more details, please take a look at the [HybridModel design doc](https://github.com/NVIDIA/Megatron-LM/issues/4620).
**How can I migrate?**
Take a look at our [migration guide](https://docs.nvidia.com/megatron-core/developer-guide/nightly/user-guide/hybrid-model-migration.html).
**Can I use an existing `GPTModel` checkpoint to train a `HybridModel`?**
Yes! You can either convert the checkpoint ahead of time or let the pretraining script automatically convert your existing `GPTModel` checkpoint to a `HybridModel` at runtime by pointing the `--pretrained-checkpoint` argument to the old `GPTModel` checkpoint when starting a training run with `pretrain_hybrid.py`.
**`HybridModel` is missing a feature I need. What should I do?**
All commonly used features supported in `GPTModel` are also supported by `HybridModel`. If there is a feature that we missed, please create a new issue..
**When will `GPTModel` be deleted?**
We do not currently have a set-date for removal; that being said, we will not remove `GPTModel` until all our users have migrated. Until then, we will continue to run CI for `GPTModel`, although test coverage will be gradually reduced over time.
Contributor guide
Research direction
This is a deprecation announcement rather than an implementation task; it names no repository files or tests. Read the linked HybridModel design document and migration guide first, then inspect the mentioned pretrain_hybrid.py entry point; the issue provides no specific code change or completion criterion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 15/100