Prevent non-colocated Megatron Inference from doing model conversion twice
Open
enhancement
Feature
- Dominant language
- Python
- Stars
- 2k
- Forks
- 561
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 145
Description
**Is your feature request related to a problem? Please describe.**
#2355 updated Megatron Inference to be a full peer of vllm inference.
Model conversion still happens twice, once on training workers, once on inference workers.
It should happen only once, and just be streamed across.
**Describe the solution you'd like**
See the comment at https://github.com/NVIDIA-NeMo/RL/pull/2355#discussion_r3313745480
**Describe alternatives you've considered**
N/A
**Additional context**
See the comment at https://github.com/NVIDIA-NeMo/RL/pull/2355#discussion_r3313745480
Contributor guide
Assessment
This issue has not been assessed yet.