huggingface / huggingface/trl

Logging how vllm importance ratios are truncated/masked in GRPOTrainer

Open
#5,231 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
19.3k
Forks
3k
Avg merge
1d 20h
Merged PRs (30d)
194

Description

### Feature request

I think logging more values related to vllm importance ratios will be helpful, e.g. logging max/min/mean of raw importance ratios (before truncation/mask) and the fraction of importance samples being truncated/masked.

### Motivation

As of now, all the logged values related to vllm importance sampling correction are based on already truncated/masked values, so it is hard to see what the raw values look like and how many importance ratios are truncated/masked.

Where the importance ratios are truncated/masked:
https://github.com/huggingface/trl/blob/1850da5a3cbaeb043b483ae78986c43246501d40/trl/trainer/grpo_trainer.py#L1807-L1814

Where the values are logged:
https://github.com/huggingface/trl/blob/1850da5a3cbaeb043b483ae78986c43246501d40/trl/trainer/grpo_trainer.py#L1955-L1977

### Your contribution

~I can send a PR to add metrics using importance ratios before truncation/masking.~
I made a PR: #5243

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.