[DGX Station][Inference] Nemotron Ultra FP8 KV scaling factors are not loaded and fall back to 1.0
- Dominant language
- TypeScript
- Stars
- 22.5k
- Forks
- 3.1k
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 715
Description
## Description
The v0.0.97 dual DGX Station inference recipe enables FP8 KV cache for Nemotron Ultra, but vLLM does not load the checkpoint `k_scale` and `v_scale` tensors for the attention layers. Both pipeline workers fall back to scaling factor 1.0 and warn that accuracy may be affected.
Platform scope: Reproduced on dual DGX Station only; other platforms not tested
Regression: Unknown — earlier versions not tested
## Environment
```text
Device: Dual DGX Station (GB300)
OS: Ubuntu 24.04.4 LTS
Architecture: aarch64
Node.js: v22.23.2
npm: 10.9.8
Docker: 29.2.1
OpenShell CLI: 0.0.85
NemoClaw: v0.0.97
OpenClaw: 2026.7.1 (2d2ddc4)
vLLM: 0.25.1
Model: nemotron-ultra
```
## Steps to Reproduce
1. Prepare two DGX Stations using the supported dual-Station fabric setup.
2. From Station A, install v0.0.97 with `NEMOCLAW_DGX_STATION_PEER` set to Station B.
3. Allow the distributed Nemotron Ultra vLLM service to start.
4. Inspect the vLLM startup logs on both pipeline workers.
## Expected Result
Because the v0.0.97 recipe enables FP8 KV cache, vLLM should load the checkpoint `k_scale` and `v_scale` tensors for every attention layer and use calibrated scaling factors.
## Actual Result
For each attention layer, vLLM finds checkpoint scale tensors under names such as `layers.7.mixer.attn.k_scale` but expects names such as `layers.7.mixer.attn.attn.k_scale`. The checkpoint scales are not loaded. Both pipeline workers then use KV cache scaling factor 1.0 and uncalibrated q/prob scaling factors.
## Logs
```text
Found k_scale in the checkpoint (e.g. layers.7.mixer.attn.k_scale), but not found the expected name in the model (e.g. layers.7.mixer.attn.attn.k_scale). k_scale is not loaded.
Found v_scale in the checkpoint (e.g. layers.7.mixer.attn.v_scale), but not found the expected name in the model (e.g. layers.7.mixer.attn.attn.v_scale). v_scale is not loaded.
Checkpoint does not provide a q scaling factor. Setting it to k_scale.
Using KV cache scaling factor 1.0 for fp8_e4m3. If this is unintended, verify that k/v_scale scaling factors are properly set in the checkpoint.
Using uncalibrated q_scale 1.0 and/or prob_scale 1.0 with fp8 attention. This may cause accuracy issues. Please make sure q/prob scaling factors are available in the fp8 checkpoint.
```
Contributor guide
Assessment
This issue has not been assessed yet.