NVIDIA-NeMo / NVIDIA-NeMo/RL

Prevent non-colocated Megatron Inference from doing model conversion twice

Open
#2,615 0 comments 0 reactions 1 assignee Claimed by @terrykong View on GitHub
enhancement Feature
Dominant language
Python
Stars
2k
Forks
561
Avg merge
4d 5h
Merged PRs (30d)
145

Description

**Is your feature request related to a problem? Please describe.**
#2355 updated Megatron Inference to be a full peer of vllm inference.

Model conversion still happens twice, once on training workers, once on inference workers.

It should happen only once, and just be streamed across.

**Describe the solution you'd like**
See the comment at https://github.com/NVIDIA-NeMo/RL/pull/2355#discussion_r3313745480

**Describe alternatives you've considered**
N/A

**Additional context**
See the comment at https://github.com/NVIDIA-NeMo/RL/pull/2355#discussion_r3313745480

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.