NVIDIA-NeMo / NVIDIA-NeMo/RL

Speedup quantization in refit for FP8 GRPO

Open
#1,467 0 comments 0 reactions 0 assignees Claimed by @Jianbing-D View on GitHub
deepseek Low Precision Performance
Dominant language
Python
Stars
2k
Forks
561
Avg merge
4d 5h
Merged PRs (30d)
145

Description

**Is your feature request related to a problem? Please describe.**
Track the effort of accelerating the in-flight quantization in refit when vllm uses FP8 precision weights.

**Describe the solution you'd like**
Quantize before transfer and do weight transfer in FP8 format; Currently we are quantizing after transfer and transfer is in BF16 format

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.