NVIDIA / NVIDIA/Megatron-LM

[Deprecation] `GPTModel`

Open
#6,071 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 3h
Merged PRs (30d)
272

Description

We are deprecating `GPTModel` in favor of `HybridModel`. **We will only be accepting critical bug fixes for `GPTModel`.**

## FAQs

**Why `HybridModel`?**

1. HybridModel is purpose-built for heterogeneity
2. It supports arbitrary layer placements, SSM layers, and flexible VPP
3. Architecture is more user-friendly, customizable, and intuitive
4. Large reduction in complexity relative to GPTModel

For more details, please take a look at the [HybridModel design doc](https://github.com/NVIDIA/Megatron-LM/issues/4620).

**How can I migrate?**

Take a look at our [migration guide](https://docs.nvidia.com/megatron-core/developer-guide/nightly/user-guide/hybrid-model-migration.html).

**Can I use an existing `GPTModel` checkpoint to train a `HybridModel`?**

Yes! You can either convert the checkpoint ahead of time or let the pretraining script automatically convert your existing `GPTModel` checkpoint to a `HybridModel` at runtime by pointing the `--pretrained-checkpoint` argument to the old `GPTModel` checkpoint when starting a training run with `pretrain_hybrid.py`.

**`HybridModel` is missing a feature I need. What should I do?**

All commonly used features supported in `GPTModel` are also supported by `HybridModel`. If there is a feature that we missed, please create a new issue..

**When will `GPTModel` be deleted?**

We do not currently have a set-date for removal; that being said, we will not remove `GPTModel` until all our users have migrated. Until then, we will continue to run CI for `GPTModel`, although test coverage will be gradually reduced over time.

Contributor guide

Open the contributing guide

Research direction

This is a deprecation announcement rather than an implementation task; it names no repository files or tests. Read the linked HybridModel design document and migration guide first, then inspect the mentioned pretrain_hybrid.py entry point; the issue provides no specific code change or completion criterion.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.