huggingface / huggingface/trl

RDPO support in GRPOTRainer

Open
#5,886 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
19.3k
Forks
3k
Avg merge
1d 20h
Merged PRs (30d)
194

Description

### Method description

RDPO (https://arxiv.org/abs/2605.13641) is a direct successor to GDPO that addresses two failure modes left unresolved by normalize_then_sum: prompt-level advantage domination under skewed/binary reward distributions, and cross-dimensional reward correlation. It replaces Z-score normalization with Magnitude-Aware Quantile normalization and adds Mahalanobis whitening within each active reward subspace before aggregation.

I am working on a problem with a few different reward directions, and RDPO seems like a better fit than GDPO.

Would a new multi_objective_aggregation="rdpo" mode within GRPOTrainer be welcome? Happy to contribute a prototype PR.

### Open source status

- [x] The method implementation is available
- [ ] The model weights are available
- [ ] The training datasets are available

### Provide useful links for the implementation

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.