NVIDIA-NeMo / NVIDIA-NeMo/RL

TRT-LLM nightly recipes fail at startup: missing tensorrt_llm module and missing HF model in CI cache

Open
#3,411 0 comments 0 reactions 1 assignee Claimed by @kajalj22 View on GitHub
Automation bug
Dominant language
Python
Stars
2k
Forks
561
Avg merge
4d 5h
Merged PRs (30d)
145

Description

Both TRT-LLM recipes in `nightly_gb200.txt` fail at startup, for **two different reasons**. Neither looks related to any particular code change — both are environment/runtime-path gaps — but nothing currently catches them, so filing for follow-up.

Found while validating #3280 (vLLM 0.20.0 → 0.25.1). These two recipes were the only entries in `nightly_gb200.txt` absent from that PR's results tables (30 entries, 28 rows), which is what prompted running them.

## The two failures

Run: pipeline `nemo-ci#59901514`.

### 1. `grpo-qwen3-1.7b-2n4g-fsdp2-trtllm` — `tensorrt_llm` not importable

```
ModuleNotFoundError: No module named 'tensorrt_llm'
ERROR SUMMARY / Error Type : Import Error
```

The TRT-LLM worker venv is built at runtime by `uv run --locked --extra trtllm --directory ` (`nemo_rl/distributed/virtual_cluster.py:80`). `tensorrt-llm` is a path source with a custom PEP 517 backend (`3rdparty/TensorRT-LLM-workspace`), so that sync has to **compile it from source** unless a prebuilt wheel is found.

The wheel cache is only wired up in the unit shard, not in the recipes:

```bash
# tests/unit/L0_Unit_Tests_Trtllm.sh
TRTLLM_WHEEL_CACHE_DIR=/opt/trtllm_wheels \
TRTLLM_REQUIRE_CACHED_WHEEL=1 \
uv run --extra trtllm bash -x ./tests/run_unit.sh ...
```

with the comment *"The release image persists the build-time wheel at this path. Require it so this shard fails quickly instead of silently starting a source compilation."*

Neither `tests/test_suites/llm/grpo-qwen3-1.7b-2n4g-fsdp2-trtllm.sh` nor the Slurm launch path sets either variable. **Plausible fix: propagate `TRTLLM_WHEEL_CACHE_DIR` (and probably `TRTLLM_REQUIRE_CACHED_WHEEL=1`) into the recipe/worker environment**, so the runtime venv reuses the image's wheel instead of attempting a source build. Worth confirming that is all it is before doing anything more elaborate.

### 2. `grpo-qwen2.5-0.5b-1n4g-megatron-trtllm-noncolocated-async` — model missing from the CI HF cache

```
OSError: We couldn't connect to 'https://huggingface.co' to load the files,
and couldn't find them in the cached files.
```

raised from `transformers/utils/hub.py::cached_files`. The recipe uses `model_name: Qwen/Qwen2.5-0.5B`; CI nodes are offline, so the model must already be present in the shared `HF_HOME`. It is not. **Fix: seed `Qwen/Qwen2.5-0.5B` into the GB200 CI HF cache.**

## Why this is probably not a regression

- Neither recipe is touched by #3280 — `git log origin/main..HEAD` on both is empty — and neither is `virtual_cluster.py:80` nor the unit shard script.
- I searched the last 25 nightly CI pipelines (including their child and grandchild pipelines) for these two job names. **The only occurrences are the run above.** They do not appear in recent nightlies at all, so there is no green baseline anywhere to compare against.

That last point is the more interesting finding: these recipes are listed in `nightly_gb200.txt` but have effectively not been exercised, so both failure modes could have been present for some time.

## Suggested follow-up

- [ ] Confirm whether propagating the TRT-LLM wheel-cache env vars is sufficient for (1)
- [ ] Seed `Qwen/Qwen2.5-0.5B` into the GB200 CI HF cache for (2)
- [ ] Work out why these two recipes are not running in nightlies despite being listed
- [ ] Consider a control run on `main` to confirm both predate the vLLM bump

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.