vllm generation with fp8 fails on gb200 and h100
- Dominant language
- Python
- Stars
- 2k
- Forks
- 561
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 145
Description
**Describe the bug**
When enabling the GB200 github CI runners, we encountered an error with vllm generation with fp8.
https://github.com/NVIDIA-NeMo/RL/actions/runs/22624061585/job/65594388506#step:3:4058
```
(VllmAsyncGenerationWorker pid=35674) (EngineCore_DP0 pid=36006) ERROR 03-03 17:37:04 [core.py:842] process_weights_after_loading(model, model_config, target_device)
(VllmAsyncGenerationWorker pid=35674) (EngineCore_DP0 pid=36006) ERROR 03-03 17:37:04 [core.py:842] File "/opt/nemo-rl/nemo_rl/models/generation/vllm/quantization/fp8.py", line 517, in process_weights_after_loading [repeated 2x across cluster]
(VllmAsyncGenerationWorker pid=35674) (EngineCore_DP0 pid=36006) ERROR 03-03 17:37:04 [core.py:842] quant_method.process_weights_after_loading(module)
(VllmAsyncGenerationWorker pid=35674) (EngineCore_DP0 pid=36006) ERROR 03-03 17:37:04 [core.py:842] maybe_post_process_fp8_weight_block(layer)
(VllmAsyncGenerationWorker pid=35674) (EngineCore_DP0 pid=36006) ERROR 03-03 17:37:04 [core.py:842] File "/opt/nemo-rl/nemo_rl/models/generation/vllm/quantization/fp8.py", line 490, in maybe_post_process_fp8_weight_block
(VllmAsyncGenerationWorker pid=35674) (EngineCore_DP0 pid=36006) ERROR 03-03 17:37:04 [core.py:842] layer.weight_scale.data.copy_(dg_weight_scale)
(VllmAsyncGenerationWorker pid=35674) (EngineCore_DP0 pid=36006) ERROR 03-03 17:37:04 [core.py:842] RuntimeError: The size of tensor a (8) must match the size of tensor b (2) at non-singleton dimension 1
```
**Steps/Code to reproduce bug**
Run the tests/unit/models/generation/test_vllm_generation.py tests on GB200 and comment out the pytest.skip that skips these tests on GB200:
- test_vllm_generation_with_hf_training_colocated
- test_vllm_generation_with_hf_training_non_colocated
- test_vllm_weight_update_and_prefix_cache_reset
- test_vllm_generation_with_megatron_training
- test_vllm_generation_with_megatron_training_moe_model
A helpful guide on on how to craft a minimal bug report http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports.
**Expected behavior**
The test_vllm_generation unit tests should all pass on GB200
**Additional context**
Add any other context about the problem here.
Contributor guide
Assessment
This issue has not been assessed yet.