NVIDIA / NVIDIA/NemoClaw

[DGX Station][Inference] Nemotron Ultra FP8 KV scaling factors are not loaded and fall back to 1.0

Open
#7,899 1 comment 0 reactions 1 assignee Claimed by @senthilr-nv View on GitHub
area: inference area: local-models NV QA platform: arm64 platform: dgx-station provider: vllm v0.0.127
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Description

The v0.0.97 dual DGX Station inference recipe enables FP8 KV cache for Nemotron Ultra, but vLLM does not load the checkpoint `k_scale` and `v_scale` tensors for the attention layers. Both pipeline workers fall back to scaling factor 1.0 and warn that accuracy may be affected.

Platform scope: Reproduced on dual DGX Station only; other platforms not tested

Regression: Unknown — earlier versions not tested

## Environment

```text
Device: Dual DGX Station (GB300)
OS: Ubuntu 24.04.4 LTS
Architecture: aarch64
Node.js: v22.23.2
npm: 10.9.8
Docker: 29.2.1
OpenShell CLI: 0.0.85
NemoClaw: v0.0.97
OpenClaw: 2026.7.1 (2d2ddc4)
vLLM: 0.25.1
Model: nemotron-ultra
```

## Steps to Reproduce

1. Prepare two DGX Stations using the supported dual-Station fabric setup.
2. From Station A, install v0.0.97 with `NEMOCLAW_DGX_STATION_PEER` set to Station B.
3. Allow the distributed Nemotron Ultra vLLM service to start.
4. Inspect the vLLM startup logs on both pipeline workers.

## Expected Result

Because the v0.0.97 recipe enables FP8 KV cache, vLLM should load the checkpoint `k_scale` and `v_scale` tensors for every attention layer and use calibrated scaling factors.

## Actual Result

For each attention layer, vLLM finds checkpoint scale tensors under names such as `layers.7.mixer.attn.k_scale` but expects names such as `layers.7.mixer.attn.attn.k_scale`. The checkpoint scales are not loaded. Both pipeline workers then use KV cache scaling factor 1.0 and uncalibrated q/prob scaling factors.

## Logs

```text
Found k_scale in the checkpoint (e.g. layers.7.mixer.attn.k_scale), but not found the expected name in the model (e.g. layers.7.mixer.attn.attn.k_scale). k_scale is not loaded.
Found v_scale in the checkpoint (e.g. layers.7.mixer.attn.v_scale), but not found the expected name in the model (e.g. layers.7.mixer.attn.attn.v_scale). v_scale is not loaded.
Checkpoint does not provide a q scaling factor. Setting it to k_scale.
Using KV cache scaling factor 1.0 for fp8_e4m3. If this is unintended, verify that k/v_scale scaling factors are properly set in the checkpoint.
Using uncalibrated q_scale 1.0 and/or prob_scale 1.0 with fp8 attention. This may cause accuracy issues. Please make sure q/prob scaling factors are available in the fp8 checkpoint.
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.