[DGX Station][Inference] Nemotron 3 Ultra vLLM hangs in flashinfer fp8_gemm autotuner per official StationDeploymentGuide command, blocking local-vLLM inference-switch verification
- Dominant language
- TypeScript
- Stars
- 22.5k
- Forks
- 3.1k
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 715
Description
## Description
When starting NVIDIA Nemotron 3 Ultra 550B-A55B-NVFP4 via vLLM using the exact docker command documented in the official NVIDIA Nemotron StationDeploymentGuide (single-host GB300 DGX Station), the vLLM engine hangs indefinitely inside the flashinfer `fp8_gemm` autotuner phase. GPU utilization stays high (97-99%) but the container's log output stops growing entirely, confirmed reproducible across two independent attempts (including one retry with `VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0` added), both hanging at the identical point in startup.
This blocks a NemoClaw manual test scenario that verifies switching a sandbox's inference route from the managed DeepSeek V4 Flash vLLM to a user-run local vLLM serving Nemotron 3 Ultra: the switch logic itself (NemoClaw's `inference set --provider vllm-local`) could not be exercised because the external Ultra vLLM never reaches a ready state.
**Platform scope:** Reproduced on DGX Station (GB300) aarch64 only; other platforms not tested.
**Regression:** Unknown — earlier versions not tested for this specific scenario.
**OpenShell issue:** No — root cause is in vLLM v0.22.0's flashinfer autotuner (or the guide's launch flags) on this hardware, not in NemoClaw or OpenShell code.
## Environment
```text
Device: DGX Station (GB300)
OS: Ubuntu 24.04.4 LTS
Architecture: aarch64
Node.js: v22.23.2
npm: 10.9.8
Docker: 29.2.1
OpenShell CLI: 0.0.106
NemoClaw: v0.0.114
OpenClaw: 2026.7.1 (2d2ddc4)
```
## Steps to Reproduce
1. On a DGX Station GB300, stop any managed vLLM occupying port 8000.
2. Pull `vllm/vllm-openai:v0.22.0` and run it per the official Nemotron 3 Ultra StationDeploymentGuide, serving `nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4` on `0.0.0.0:8000` with `--cpu-offload-gb 150 --cpu-offload-params experts --gpu-memory-utilization 0.9 --max-num-batched-tokens 8192` (plus `--max-num-seqs 680`, a required correction since the guide's command as published omits this and otherwise fails immediately with a Mamba-cache-blocks ValueError).
3. Watch container logs and GPU utilization.
## Expected Result
The vLLM engine completes startup within the documented window and serves an OpenAI-compatible API; `curl http://localhost:8000/v1/models` lists the Ultra model id.
## Actual Result
The engine enters the flashinfer `fp8_gemm` autotuning phase after KV cache setup and never progresses further. Confirmed via two independent 15-20 second interval checks: `docker logs` line count and tail content are byte-identical across checks despite GPU utilization remaining at 97-99%, meaning the process is spinning without making forward progress. `curl http://localhost:8000/v1/models` returns empty/connection-refused throughout. Reproduced twice: once with default CUDA graph memory profiling, once with `VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0` added — both hang at the identical log position.
## Logs
```text
Last progressing log lines before the hang (identical across both attempts' final state):
INFO kv_cache_utils.py:1733 GPU KV cache size: 8,912,896 tokens (or 9,096,396 with profiling disabled)
INFO kv_cache_utils.py:1734 Maximum concurrency for 262,144 tokens per request: 34.00x (or 34.70x)
INFO autotuner.py:615 flashinfer.jit: [Autotuner]: Autotuning process starts ...
[AutoTuner]: Tuning fp8_gemm: 100%|##########| 21/21 [00:21
```
Contributor guide
Research direction
Reproduce the official Nemotron StationDeploymentGuide Docker command on a DGX Station GB300, including the documented flags and the added --max-num-seqs 680. Inspect the container logs around flashinfer's fp8_gemm autotuner and compare runs with and without VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0. Done means identifying an upstream vLLM/flashinfer or launch-flag resolution so the model reaches /v1/models; no NemoClaw code path is implicated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker
- Domain
- ai, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 25/100