feat: Megatron Inference lacks `defer_model_load`
- Dominant language
- Python
- Stars
- 2k
- Forks
- 561
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 145
Description
**Is your feature request related to a problem? Please describe.**
Currently, Megatron Inference lacks `defer_model_load`, making the deferred load code complicated; see #3569.
**Describe the solution you'd like**
When Megatron Inference becomes capable of doing `defer_model_load`, the logic introduced by #3569 can be significantly simplified.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Add any other context or screenshots about the feature request here.
Contributor guide
Research direction
Start by reading the deferred-load logic introduced by #3569 and the Megatron Inference entry points. Determine how Megatron Inference should support `defer_model_load`; done when that capability works and the logic from #3569 can be significantly simplified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100