Logging how vllm importance ratios are truncated/masked in GRPOTrainer
- Dominant language
- Python
- Stars
- 19.3k
- Forks
- 3k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 194
Description
### Feature request
I think logging more values related to vllm importance ratios will be helpful, e.g. logging max/min/mean of raw importance ratios (before truncation/mask) and the fraction of importance samples being truncated/masked.
### Motivation
As of now, all the logged values related to vllm importance sampling correction are based on already truncated/masked values, so it is hard to see what the raw values look like and how many importance ratios are truncated/masked.
Where the importance ratios are truncated/masked:
https://github.com/huggingface/trl/blob/1850da5a3cbaeb043b483ae78986c43246501d40/trl/trainer/grpo_trainer.py#L1807-L1814
Where the values are logged:
https://github.com/huggingface/trl/blob/1850da5a3cbaeb043b483ae78986c43246501d40/trl/trainer/grpo_trainer.py#L1955-L1977
### Your contribution
~I can send a PR to add metrics using importance ratios before truncation/masking.~
I made a PR: #5243
Contributor guide
Assessment
This issue has not been assessed yet.