Numerical stability in KL's exp function
Open
- Dominant language
- Python
- Stars
- 19.3k
- Forks
- 3k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 194
Description
I experienced some numerical issues here
https://github.com/huggingface/trl/blob/e3244d2d096ff1e2e248c931d06d39e165e20623/trl/trainer/grpo_trainer.py#L943
My `ref_per_token_logps` and `per_token_logps` differ more during training, leading to a large value in `exp` and getting `inf` on the step.
Contributor guide
Research direction
Start with trl/trainer/grpo_trainer.py at line 943 and inspect how the differing ref_per_token_logps and per_token_logps reach exp during training. Reproduce the described case where the value becomes inf, then verify the numerical-stability change prevents that result without changing normal training behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100