Packing sequences for memory efficiency in GRPO and other preference learning implementations
- Dominant language
- Python
- Stars
- 19.3k
- Forks
- 3k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 194
Description
### Feature request
Similar to packing in SFT, we can apply packing to sequences in different generations of the same group. Sharing the prefix should save memory, especially in situations where prefix length >> completion length.
Some reference implementations:
- https://github.com/frankxwang/dpo-prefix-sharing
- https://github.com/li-plus/flash-preference/tree/main
### Motivation
Many classes of problems involving large contexts end up being dominated by prompt length. This should help with memory savings in those situations
### Your contribution
I would love to take a stab at a PR, but want to discuss the feasibility and interest in adding support for this in TRL
Contributor guide
Assessment
This issue has not been assessed yet.