RDPO support in GRPOTRainer
- Dominant language
- Python
- Stars
- 19.3k
- Forks
- 3k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 194
Description
### Method description
RDPO (https://arxiv.org/abs/2605.13641) is a direct successor to GDPO that addresses two failure modes left unresolved by normalize_then_sum: prompt-level advantage domination under skewed/binary reward distributions, and cross-dimensional reward correlation. It replaces Z-score normalization with Magnitude-Aware Quantile normalization and adds Mahalanobis whitening within each active reward subspace before aggregation.
I am working on a problem with a few different reward directions, and RDPO seems like a better fit than GDPO.
Would a new multi_objective_aggregation="rdpo" mode within GRPOTrainer be welcome? Happy to contribute a prototype PR.
### Open source status
- [x] The method implementation is available
- [ ] The model weights are available
- [ ] The training datasets are available
### Provide useful links for the implementation
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.