NVIDIA / NVIDIA/NemoClaw

[DGX Station][Inference] Nemotron 3 Ultra vLLM hangs in flashinfer fp8_gemm autotuner per official StationDeploymentGuide command, blocking local-vLLM inference-switch verification

Open
#10,108 0 comments 0 reactions 0 assignees View on GitHub
area: inference area: local-models area: providers area: routing NV QA platform: arm64 platform: dgx-station provider: vllm
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Description

When starting NVIDIA Nemotron 3 Ultra 550B-A55B-NVFP4 via vLLM using the exact docker command documented in the official NVIDIA Nemotron StationDeploymentGuide (single-host GB300 DGX Station), the vLLM engine hangs indefinitely inside the flashinfer `fp8_gemm` autotuner phase. GPU utilization stays high (97-99%) but the container's log output stops growing entirely, confirmed reproducible across two independent attempts (including one retry with `VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0` added), both hanging at the identical point in startup.

This blocks a NemoClaw manual test scenario that verifies switching a sandbox's inference route from the managed DeepSeek V4 Flash vLLM to a user-run local vLLM serving Nemotron 3 Ultra: the switch logic itself (NemoClaw's `inference set --provider vllm-local`) could not be exercised because the external Ultra vLLM never reaches a ready state.

**Platform scope:** Reproduced on DGX Station (GB300) aarch64 only; other platforms not tested.
**Regression:** Unknown — earlier versions not tested for this specific scenario.
**OpenShell issue:** No — root cause is in vLLM v0.22.0's flashinfer autotuner (or the guide's launch flags) on this hardware, not in NemoClaw or OpenShell code.

## Environment

```text
Device: DGX Station (GB300)
OS: Ubuntu 24.04.4 LTS
Architecture: aarch64
Node.js: v22.23.2
npm: 10.9.8
Docker: 29.2.1
OpenShell CLI: 0.0.106
NemoClaw: v0.0.114
OpenClaw: 2026.7.1 (2d2ddc4)
```

## Steps to Reproduce

1. On a DGX Station GB300, stop any managed vLLM occupying port 8000.
2. Pull `vllm/vllm-openai:v0.22.0` and run it per the official Nemotron 3 Ultra StationDeploymentGuide, serving `nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4` on `0.0.0.0:8000` with `--cpu-offload-gb 150 --cpu-offload-params experts --gpu-memory-utilization 0.9 --max-num-batched-tokens 8192` (plus `--max-num-seqs 680`, a required correction since the guide's command as published omits this and otherwise fails immediately with a Mamba-cache-blocks ValueError).
3. Watch container logs and GPU utilization.

## Expected Result

The vLLM engine completes startup within the documented window and serves an OpenAI-compatible API; `curl http://localhost:8000/v1/models` lists the Ultra model id.

## Actual Result

The engine enters the flashinfer `fp8_gemm` autotuning phase after KV cache setup and never progresses further. Confirmed via two independent 15-20 second interval checks: `docker logs` line count and tail content are byte-identical across checks despite GPU utilization remaining at 97-99%, meaning the process is spinning without making forward progress. `curl http://localhost:8000/v1/models` returns empty/connection-refused throughout. Reproduced twice: once with default CUDA graph memory profiling, once with `VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0` added — both hang at the identical log position.

## Logs

```text
Last progressing log lines before the hang (identical across both attempts' final state):
INFO kv_cache_utils.py:1733 GPU KV cache size: 8,912,896 tokens (or 9,096,396 with profiling disabled)
INFO kv_cache_utils.py:1734 Maximum concurrency for 262,144 tokens per request: 34.00x (or 34.70x)
INFO autotuner.py:615 flashinfer.jit: [Autotuner]: Autotuning process starts ...
[AutoTuner]: Tuning fp8_gemm: 100%|##########| 21/21 [00:21
```

Contributor guide

Open the contributing guide

Research direction

Reproduce the official Nemotron StationDeploymentGuide Docker command on a DGX Station GB300, including the documented flags and the added --max-num-seqs 680. Inspect the container logs around flashinfer's fp8_gemm autotuner and compare runs with and without VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0. Done means identifying an upstream vLLM/flashinfer or launch-flag resolution so the model reaches /v1/models; no NemoClaw code path is implicated.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker
Domain
ai, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.