vLLM 0.25 bump: cleanup and loose ends

Tracking issue for follow-ups deliberately deferred out of #3280 (vLLM 0.20.0 → 0.25.1) so the bump itself stays reviewable. None of these block that PR; each is either pre-existing, upstream, or needs work that does not belong in a version bump.

---

## 1. PPO critic grad-norm regression on 0.25 — **highest priority**

`tests/functional/ppo_automodel.sh` shows a large, measured change in a **training-side** metric after the bump:

| metric (`ppo_automodel`) | vLLM 0.20 | vLLM 0.25 | bound | change |
|---|---|---|---|---|
| `token_mult_prob_error` (max) | 1.0263 | 1.0197 | < 1.05 | ≈ same |
| `probs_ratio_clamped_min` | 0.8000 | 0.8000 | — | identical |
| `probs_ratio_clamped_max` (max) | 1.2800 | 1.2800 | < 1.29 | identical |
| `critic/loss` (max) | 4.037 | 3.087 | < 6.0 | lower |
| `critic/loss` (min) | 2.160 | 1.601 | ≥ 0 | lower |
| **`critic/explained_var` (max)** | **−0.7208** | **−5.2847** | ≤ 1.0001 | **7.3× worse** |
| **`critic/grad_norm` (max)** | **110.54** | **700.22** | < 350 | **6.3× higher** |

Sources: 0.25 from #3280's `L1_Functional_Tests_PPO` (job `90199439799`); 0.20 baseline from #3360 (job `90219748938`), which passes at 110.54.

**What the data says.** Every policy-side metric is unchanged — rollout logprobs still agree with training logprobs as well as they did on 0.20 — so this is not generation being broken in any way this test detects. The critic is where it diverges: `explained_var` at −5.28 means the value function fits the returns much worse than predicting their mean, and a badly-fitting critic produces large value errors and therefore large gradients. `explained_var` and `grad_norm` are most likely one phenomenon seen twice.

**The question to chase:** why is the critic's regression target harder under 0.25 when policy-side metrics are flat? Threads: do returns/advantages differ (nothing in this test asserts on advantage scale); do generation lengths/truncation differ; anything touching the **value** model rather than the policy model.

**Interim state in #3280:** the ceiling is raised so the bump is not blocked, with a comment pointing here. That is explicitly a placeholder, not a verdict — **please do not close this by declaring the raised bound correct.** If 700 turns out to be acceptable, that needs to rest on understanding the critic-fit change, and the bound should then be set from measured behaviour.

History worth knowing: an earlier revision of #3280 raised this ceiling to 1500 claiming "all observed metrics are healthy on-policy values". That claim was unsubstantiated — no functional run had executed on the PR at that point. It was reverted, which is the only reason the regression surfaced at all: 700 passes silently under 1500.

Caveats: n=1 per version, and 2 steps of a freshly-initialised value head is a high-variance regime, so the exact multipliers should not be treated as precise. Repeat runs on each side would firm this up.

- [ ] Reproduce with ≥2 runs per version
- [ ] Identify the mechanism
- [ ] Set the bound from measured behaviour, or fix the cause
- [ ] Consider adding a **floor** on `critic/explained_var` — the current assertion is `<= 1.0001`, which only catches impossible-good values, so a 7× degradation in critic fit triggers nothing on its own and is only visible via the downstream `grad_norm`

---

## 2. TRT-LLM nightly recipes — see #3411

Both `nightly_gb200.txt` TRT-LLM recipes fail at startup, for two unrelated reasons (missing `tensorrt_llm` module; missing HF model in the CI cache). Tracked separately in #3411. Almost certainly pre-existing rather than bump-related — they appear to have not been running in nightlies at all.

---

## 3. DeepSeek-V3 GB200: dies during checkpointing at step 10

`grpo-deepseek-v3-32n4g` on GB200 now gets all the way to step 10 — 22 completed refits, CUDA graphs captured, validation started — and then dies while saving a checkpoint:

```
ray.exceptions.ActorDiedError: The actor died unexpectedly before finishing this task.
Worker exit type: SYSTEM_ERROR — Worker unexpectedly exits with a connection error code 2
```

with a corresponding Slurm step failure on a named node. All three vLLM-0.25 startup blockers fixed in #3280 show zero occurrences in that run (no `EADDRINUSE`, no `ZMQError`, no refit-buffer assertion), and the H100 equivalent passes at the same step budget. Reads as node/transport-level rather than application logic, but it has not been root-caused.

- [ ] Re-run to establish whether it is reproducible or a one-off infrastructure failure

---

## 4. Upstream vLLM fixes worth contributing back

Two source patches in #3280 fix bugs that are **byte-identical on vLLM `main`**, so they affect upstream users too:

- `MessageQueue` remote-socket probe-then-bind race (`shm_broadcast.py`): every worker on a non-driver node races for the same `VLLM_PORT`. Fixed locally by retrying at the bind. Context in RL-1111.
- `RayExecutorV2._select_tcpstore_port`: the TCPStore port is chosen from the same `VLLM_PORT` scan the broadcast `MessageQueue` binds and holds. Context in RL-1104.

- [ ] File both upstream

---

## 5. Smaller loose ends

- **No nightly coverage for node-spanning engines.** Zero of the 140 nightly tests have `tensor_parallel_size * pipeline_parallel_size > cluster.gpus_per_node`, which is the only reason two separate startup bugs shipped undetected. All 17 such recipes live in performance/release suites. Both bugs now have cheap unit-level reproductions, but a ≤2-node functional test would close the class properly.
- **`test_vllm_megatron_pipeline_parallel` is ~9% slower on 0.25** (103.80s → 113.10s call time), which is why its 120s budget had to go to 240s. Not a generation-throughput regression — it is engine-init overhead in a test that builds both a Megatron pp=2 policy and a vLLM engine inside one budget. Worth knowing if init time matters elsewhere.
- **`fp8-rollouts` KV-cache pin is a ~45% cut, and CI cannot detect if it is too small.** Now quantified. Measured on `grpo-llama3.1-8b-instruct-1n8g-megatron-fp8-rollouts.v3`:

| | vLLM 0.20 (job 371232800) | 0.25 @ head, pinned |
|---|---|---|
| Available KV cache memory | **36.13 GiB** | 20.00 GiB |
| Sleep pool discarded | 36.19 GiB | 20.00 GiB |

So the 20 GiB pin is **not** a 0.20-equivalent restoration — it is the empirical value that stops the wake-up OOM. The recipe comment previously claimed the opposite ("several GiB larger", "0.20-equivalent"); corrected in 72f550c18.

The risk is that this recipe's driver asserts only `token_mult_prob_error`, with no `total_step_time` gate like the moonlight fp8-e2e recipe has — so a throughput regression from an over-tight pin passes CI silently. Worth either adding a step-time gate to this driver or re-deriving the pin from a profile.

- **Two sibling fp8 recipes fail with the same wake-up OOM and have no pin.** `llm_grpo_llama3_1_8b_instruct_2n8g_megatron_fp8_e2e` and `llm_grpo_qwen3_8b_base_1n8g_fp8_kvcache_megatron` both die with `CUDA Error: out of memory at cumem_allocator.cpp` — on **0.20 (`:139`) as well as 0.25 (`:163`)**, so this is pre-existing and not caused by the bump. They keep `gpu_memory_utilization: 0.4` with no `kv_cache_memory_bytes`, and `0.4` is therefore **not** demonstrated to be safe; they simply never get far enough to prove otherwise.

The pin is known to work on this failure class — it took `fp8-rollouts` from red-on-main to green-at-head. Deliberately not applied to these two in #3280, because the `fp8-rollouts` value came from a measured 0.20 figure and there is no equivalent for a 2n8g run or a different model without profiling each.
- **`_patch_vllm_init_workers_ray` is inert under the 0.25 default.** `VLLM_USE_RAY_V2_EXECUTOR_BACKEND` flipped `"0"` → `"1"`, so `RayExecutorV2` is selected and it has no `_init_workers_ray`; workers get the right interpreter from Ray's `runtime_env` inheritance instead. The patch is kept because it is still load-bearing when the v1 executor is selected explicitly, and its logging now says so — but it is a candidate for deletion once the v1 executor path is confirmed dead.

This tracking issue contains several independent investigations; start with tests/functional/ppo_automodel.sh and reproduce the vLLM 0.20 versus 0.25 critic metrics across at least two runs. For upstream work, read shm_broadcast.py and RayExecutorV2._select_tcpstore_port. Done means the regression mechanism is identified and the bound is measured or the cause fixed, with the remaining checklist items separately resolved.

Written by the indexing model from the issue text.