linkedin / linkedin/Liger-Kernel
Generalized PPO loss (& improve current GRPO loss)
Open
@qingquansong is already working on this.
Since Mar 24, 2025.
- Dominant language
- Python
- Stars
- 6.6k
- Forks
- 603
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 47
Description
🚀 The feature, motivation and pitch
- 1. Current GRPO assume KL term is added and advantage is computed inside the loss, we wanna open this to become configurable by user
- 2. Clipping is not used in GRPO and need to be added (also as an option for generalized PPO case)
- 3. Reference model logits/prob can be provided directly without providing LM head + hidden states as an option
- 4. Old policy prob or LM head/hidden states is not provided for importance sampling purpose in GRPO.
Alternatives
No response
Additional context
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.