NVIDIA-NeMo / NVIDIA-NeMo/RL

Support use vLLM DP + EP in async engine

Open
#1,101 3 comments 0 reactions 2 assignees Claimed by @guyueh1 View on GitHub
Feature Speed vllm
Dominant language
Python
Stars
2k
Forks
561
Avg merge
4d 5h
Merged PRs (30d)
145

Description

https://github.com/NVIDIA-NeMo/RL/pull/1081 supports using vLLM DP + EP in sync engine.

vLLM's DP + EP is supported for async engine for serve mode, but has issues for offline mode (which we used).
We need to hack something in vLLM to support that.

Ideally, after vLLM's FP8 inference for MoE model and DeepEP supported, we can get some speedup from DP + EP, and we need to bring the benefit to async RL / async rollout which needs to use vLLM async engine.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.