NVIDIA-NeMo / NVIDIA-NeMo/RL

feat: Megatron Inference lacks `defer_model_load`

Open
#3,752 0 comments 0 reactions 0 assignees View on GitHub
enhancement Feature
Dominant language
Python
Stars
2k
Forks
561
Avg merge
4d 5h
Merged PRs (30d)
145

Description

**Is your feature request related to a problem? Please describe.**
Currently, Megatron Inference lacks `defer_model_load`, making the deferred load code complicated; see #3569.

**Describe the solution you'd like**
When Megatron Inference becomes capable of doing `defer_model_load`, the logic introduced by #3569 can be significantly simplified.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.

Contributor guide

Open the contributing guide

Research direction

Start by reading the deferred-load logic introduced by #3569 and the Megatron Inference entry points. Determine how Megatron Inference should support `defer_model_load`; done when that capability works and the logic from #3569 can be significantly simplified.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.