NVIDIA-NeMo / NVIDIA-NeMo/RL

vllm generation with fp8 fails on gb200 and h100

Open
#2,081 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2k
Forks
561
Avg merge
4d 5h
Merged PRs (30d)
145

Description

**Describe the bug**

When enabling the GB200 github CI runners, we encountered an error with vllm generation with fp8.

https://github.com/NVIDIA-NeMo/RL/actions/runs/22624061585/job/65594388506#step:3:4058

```
(VllmAsyncGenerationWorker pid=35674) (EngineCore_DP0 pid=36006) ERROR 03-03 17:37:04 [core.py:842] process_weights_after_loading(model, model_config, target_device)
(VllmAsyncGenerationWorker pid=35674) (EngineCore_DP0 pid=36006) ERROR 03-03 17:37:04 [core.py:842] File "/opt/nemo-rl/nemo_rl/models/generation/vllm/quantization/fp8.py", line 517, in process_weights_after_loading [repeated 2x across cluster]
(VllmAsyncGenerationWorker pid=35674) (EngineCore_DP0 pid=36006) ERROR 03-03 17:37:04 [core.py:842] quant_method.process_weights_after_loading(module)
(VllmAsyncGenerationWorker pid=35674) (EngineCore_DP0 pid=36006) ERROR 03-03 17:37:04 [core.py:842] maybe_post_process_fp8_weight_block(layer)
(VllmAsyncGenerationWorker pid=35674) (EngineCore_DP0 pid=36006) ERROR 03-03 17:37:04 [core.py:842] File "/opt/nemo-rl/nemo_rl/models/generation/vllm/quantization/fp8.py", line 490, in maybe_post_process_fp8_weight_block
(VllmAsyncGenerationWorker pid=35674) (EngineCore_DP0 pid=36006) ERROR 03-03 17:37:04 [core.py:842] layer.weight_scale.data.copy_(dg_weight_scale)
(VllmAsyncGenerationWorker pid=35674) (EngineCore_DP0 pid=36006) ERROR 03-03 17:37:04 [core.py:842] RuntimeError: The size of tensor a (8) must match the size of tensor b (2) at non-singleton dimension 1
```

**Steps/Code to reproduce bug**

Run the tests/unit/models/generation/test_vllm_generation.py tests on GB200 and comment out the pytest.skip that skips these tests on GB200:
- test_vllm_generation_with_hf_training_colocated
- test_vllm_generation_with_hf_training_non_colocated
- test_vllm_weight_update_and_prefix_cache_reset
- test_vllm_generation_with_megatron_training
- test_vllm_generation_with_megatron_training_moe_model

A helpful guide on on how to craft a minimal bug report http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports.

**Expected behavior**

The test_vllm_generation unit tests should all pass on GB200

**Additional context**

Add any other context about the problem here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.