GRPOTrainer: Parallelism for updating named params
Open
✨ enhancement
🏋 GRPO
- Dominant language
- Python
- Stars
- 19.3k
- Forks
- 3k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 194
Description
### Feature request
Running GRPOTrainer with an MOE like Qwen3/Qwen-30BA3B is incredibly slow because of move_model_to_vllm which loops through all the named params and sequentially updates the vLLM server for each of the named params. Since the MOE (128 experts) has a large number of named params, this step is slow.
Curious if we can do this updated step parallely?
https://github.com/huggingface/trl/blob/main/trl/trainer/grpo_trainer.py#L917-L920
### Motivation
Better support for MOE GRPO training
### Your contribution
Can submit PR if this is a viable approach
Contributor guide
Assessment
This issue has not been assessed yet.