Generation backends disagree on how to report "no OpenAI server": [None] vs []
- Dominant language
- Python
- Stars
- 2k
- Forks
- 561
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 145
Description
## Summary
`dp_openai_server_base_urls` is meant to hold one OpenAI-compatible base URL per data-parallel rank. When a backend has no HTTP server to report, the four backends disagree about what to put there: Megatron returns an empty list, the other three return a list of `None`s.
Megatron filters the `None`s out and can then use a plain truth test:
```python
# nemo_rl/models/generation/megatron/megatron_generation.py
if (
not self.dp_openai_server_base_urls # L368 — plain truth test works
and self.cfg["mcore_generation_config"]["expose_http_server"]
):
...
self.dp_openai_server_base_urls = [
url for url in ray.get(url_futures) if url is not None # L374-376
]
```
[megatron_generation.py:368](https://github.com/NVIDIA-NeMo/RL/blob/e9a855ff4bc0a41967e9491c41de093527e123b6/nemo_rl/models/generation/megatron/megatron_generation.py#L368),
[:374-376](https://github.com/NVIDIA-NeMo/RL/blob/e9a855ff4bc0a41967e9491c41de093527e123b6/nemo_rl/models/generation/megatron/megatron_generation.py#L374-L376)
The other three keep a placeholder entry instead:
| backend | what it returns | where |
|---|---|---|
| vLLM (sync engine) | `[None]` | [vllm_generation.py:529](https://github.com/NVIDIA-NeMo/RL/blob/e9a855ff4bc0a41967e9491c41de093527e123b6/nemo_rl/models/generation/vllm/vllm_generation.py#L529) |
| vLLM (deferred load, sync engine) | `[None]` | [vllm_generation.py:548](https://github.com/NVIDIA-NeMo/RL/blob/e9a855ff4bc0a41967e9491c41de093527e123b6/nemo_rl/models/generation/vllm/vllm_generation.py#L548) |
| TRT-LLM (no HTTP server) | `[None] * dp_size` | [trtllm_generation.py:266-267](https://github.com/NVIDIA-NeMo/RL/blob/e9a855ff4bc0a41967e9491c41de093527e123b6/nemo_rl/models/generation/trtllm/trtllm_generation.py#L266-L267) |
| Dynamo (no token wrapper) | `[None]` | [dynamo_generation.py:220](https://github.com/NVIDIA-NeMo/RL/blob/e9a855ff4bc0a41967e9491c41de093527e123b6/nemo_rl/models/generation/dynamo/dynamo_generation.py#L220) |
## Why it is worth changing
The placeholder does not stand for anything. In each of those branches there is no server at all, so there is no per-rank slot to hold open — the list length carries no information. What it does do is make the value lie to a plain truth test: `[None]` is truthy, so "we have URLs" reads as true when there are none.
Callers have to know about that and work around it:
- [`_shard_base_urls`](https://github.com/NVIDIA-NeMo/RL/blob/e9a855ff4bc0a41967e9491c41de093527e123b6/nemo_rl/algorithms/single_controller_utils/setup.py#L768-L773) cannot write `if not urls`; it needs `if not any(urls)`.
- [grpo.py:1543](https://github.com/NVIDIA-NeMo/RL/blob/e9a855ff4bc0a41967e9491c41de093527e123b6/nemo_rl/algorithms/grpo.py#L1543) prints the count, so a sync run reports `Reserved 1 vLLM server URLs: [None]`.
It also forces the type to be wider than the values ever are. Every one of these lists is either all real URLs or all `None`s — never mixed. If the "none available" case returned `[]`, the type could be `list[str]` instead of `list[Optional[str]]`, and callers downstream would not have to carry the `Optional` around. This came up while reviewing #3367, where a function signature was widened to `list[Optional[str]]` to match the producer's declared type even though a `None` cannot reach that code path.
## Suggested change
Make the three backends match Megatron: return `[]` when there is no HTTP server to report, and narrow `dp_openai_server_base_urls` to `list[str]`.
Worth checking while doing it:
- [`tests/unit/models/generation/test_dynamo_generation.py:167`](https://github.com/NVIDIA-NeMo/RL/blob/e9a855ff4bc0a41967e9491c41de093527e123b6/tests/unit/models/generation/test_dynamo_generation.py#L167) asserts `== [None]` and would need updating.
- TRT-LLM's version is length `dp_size` rather than 1, so anything that reads the length in the no-server case should be checked before changing it.
- `_shard_base_urls` and the `any(...)` guards can be simplified once the sentinel is gone, but they stay correct either way, so they can be cleaned up separately.
Contributor guide
Research direction
Start with the cited no-server branches in vllm_generation.py, trtllm_generation.py, and dynamo_generation.py, then inspect dp_openai_server_base_urls callers such as _shard_base_urls and grpo.py. Update the affected tests, including tests/unit/models/generation/test_dynamo_generation.py:167, and check TRT-LLM length handling. Done means no-server cases return [] consistently and the type and callers no longer require Optional URLs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100