huggingface / huggingface/trl

Numerical stability in KL's exp function

Open
#3,015 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
19.3k
Forks
3k
Avg merge
1d 20h
Merged PRs (30d)
194

Description

I experienced some numerical issues here
https://github.com/huggingface/trl/blob/e3244d2d096ff1e2e248c931d06d39e165e20623/trl/trainer/grpo_trainer.py#L943

My `ref_per_token_logps` and `per_token_logps` differ more during training, leading to a large value in `exp` and getting `inf` on the step.

Contributor guide

Open the contributing guide

Research direction

Start with trl/trainer/grpo_trainer.py at line 943 and inspect how the differing ref_per_token_logps and per_token_logps reach exp during training. Reproduce the described case where the value becomes inf, then verify the numerical-stability change prevents that result without changing normal training behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.