NVIDIA-NeMo / NVIDIA-NeMo/Gym

feat: SGLang support

Open
#976 0 comments 0 reactions 1 assignee View on GitHub

@linnanwang is already working on this.

Since Jun 23, 2026.

models
Dominant language
Python
Stars
1.2k
Forks
349
Avg merge
1d 21h
Merged PRs (30d)
318

Description

Use cases, pain points, and background
Several RL training frameworks (e,g, VeRL, Slime) offer SGLang as a generation backend. If Gym doesn't support SGLang, users whose training loop runs on SGLang must either stand up a separate vLLM instance just for Gym, or manually proxy requests — adding latency and operational complexity. A first-class sglang_model server would let these frameworks plug into Gym directly.

Description:
Add a new responses_api_models/sglang_model/ model server that targets a remote SGLang OpenAI-compatible endpoint, analogous to the existing vllm_model

Design:
i think there are a few key requirements we would need to translate from vllm to sglang side:

  1. on policy corrections in vllm (just need to find the cleanest way to insert this logic for sglang) https://github.com/NVIDIA-NeMo/RL/blob/cd54f8f492d013f19f3bfa3af6ba5d3aab530e52/nemo_rl/models/generation/vllm/vllm_worker_async.py#L41
  2. getting the logprobs and token IDs for a chat completion request https://github.com/NVIDIA-NeMo/Gym/blob/18384bfc428045cc65751eccde7d3da7ae4db6a2/responses_api_models/vllm_model/app.py#L502
  3. applying the responses <> chat completions converter https://github.com/NVIDIA-NeMo/Gym/blob/18384bfc428045cc65751eccde7d3da7ae4db6a2/responses_api_models/vllm_model/app.py#L141
  4. max seq len hit error handling (might be specific to vllm) https://github.com/NVIDIA-NeMo/Gym/blob/18384bfc428045cc65751eccde7d3da7ae4db6a2/responses_api_models/vllm_model/app.py#L457

Out of scope:
What are some items that this issue could be mistaken to cover that this issue should explicitly NOT cover?

Acceptance Criteria:

  • sglang_model server starts, passes health checks, and serves /v1/chat/completions and /v1/responses
  • Responses ↔ ChatCompletions conversion works correctly (tool calls, reasoning, multi-turn)
  • Token ID information (prompt_token_ids, generation_token_ids, generation_log_probs) is returned when return_token_id_information: true
  • ng_collect_rollouts completes successfully against an SGLang endpoint using sglang_model
  • Unit tests pass with ≥ 95% coverage
  • Config and dogs updated
  • End-to-end RL training run (e.g. GRPO) using sglang_model converges comparably to the same run using vllm_model on the same benchmark and model checkpoint.
  • Reward curves and pass@1 metrics match within acceptable variance (< 1%) across identical ng_collect_rollouts runs against both backends.
  • Token IDs returned by SGLang are consistent with the tokenizer — round-tripping through create_tokenize, generate, and the returned prompt_token_ids / generation_token_ids produces the same token sequences as vLLM, with no re-tokenization drift.
  • Log-probs from SGLang match vLLM's to within numerical precision (fp16/bf16 tolerance) for the same prompt and greedy decoding, confirming the training signal is equivalent.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.