huggingface / huggingface/trl

`compute_metrics` in `GRPOTrainer`

Open
#2,959 3 comments 13 reactions 0 assignees View on GitHub
✨ enhancement 🏋 GRPO
Dominant language
Python
Stars
19.3k
Forks
3k
Avg merge
1d 20h
Merged PRs (30d)
194

Description

### Feature request

Can you please add `compute_metrics` in the `GRPOTrainer`?

I can see its available in other, i.e., [DPOTrainer](https://huggingface.co/docs/trl/v0.15.2/dpo_trainer#trl.DPOTrainer.compute_metrics)

### Motivation

1. We can compute accuracy or downstream eval metrics over the evaluation dataset
2. That will help in more fine-grained logging
3. Better metric for `metric_for_best_model`

### Your contribution

I am open to submit a PR

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.