NVIDIA-NeMo / NVIDIA-NeMo/RL

Add vLLM↔TRT-LLM /v1/chat/completions parity test (CI)

Open
#3,342 3 comments 0 reactions 1 assignee Claimed by @hchings View on GitHub
Feature testing
Dominant language
Python
Stars
2k
Forks
561
Avg merge
4d 5h
Merged PRs (30d)
145

Description

## Summary

The TRT-LLM OpenAI HTTP server (`nemo_rl/models/generation/trtllm/trtllm_http_server.py`) and the vLLM server independently re-implement the `/v1/chat/completions` path — prompt construction, tool parsing, reasoning parsing, and EOS-stripping — sharing only [`replace_prefix_tokens`](https://github.com/NVIDIA-NeMo/RL/blob/main/nemo_rl/models/generation/openai_server_utils.py). Nothing currently guarantees the two backends emit **identical tokens** for the same input, yet NeMo-Gym consumes their outputs interchangeably.

This was raised during review of #3130 ([thread](https://github.com/NVIDIA-NeMo/RL/pull/3130#discussion_r3620330266)). That PR added isolated TRT-LLM-side unit coverage (`tests/unit/models/generation/test_trtllm_http_server.py`) and is being merged first to unblock mlperf (validated on a qwen3.5-35B workload against vLLM); the **cross-backend parity test** is deferred to this follow-up.

## Proposed test

Add a greedy (`temperature=0.0`) parity test: the same small model (e.g. Qwen3-0.6B, TP=1) and the same multi-turn scenario driven through each backend's `/v1/chat/completions`, asserting **bitwise-identical** `prompt_token_ids`, `generation_token_ids`, and `content` (the exact fields NeMo-Gym reads back).

The scenario should include a **tool-call turn** (request with `tools=`, assistant emits a ``, then a `tool` result message and a follow-up turn) — this exercises the highest-drift surfaces at once: tool parsing/serialization, `finish_reason="tool_calls"`, and the prefix splice across the tool boundary.

Since vllm/trtllm live in separate extras and CI lanes, the practical shape is a committed golden of `(prompt_token_ids, generation_token_ids, content, tool_calls)` that each backend's lane asserts against (TRT-LLM side in `tests/unit/models/generation/trtllm/test_trtllm_http_server.py` with `pytestmark = pytest.mark.trtllm`), or a functional/L1 nightly launching both backends — which would also cover the "Add CI test(s)" TODO.

SGLang is out of scope — it's driven token-in/token-out and never hits this server layer.

## Links
- PR: #3130
- Review thread: https://github.com/NVIDIA-NeMo/RL/pull/3130#discussion_r3620330266
- Isolated coverage already added in #3130: `tests/unit/models/generation/test_trtllm_http_server.py`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.