huggingface / huggingface/trl

Packing sequences for memory efficiency in GRPO and other preference learning implementations

Open
#3,549 11 comments 1 reaction 0 assignees View on GitHub
✨ enhancement 🏋 GRPO
Dominant language
Python
Stars
19.3k
Forks
3k
Avg merge
1d 20h
Merged PRs (30d)
194

Description

### Feature request

Similar to packing in SFT, we can apply packing to sequences in different generations of the same group. Sharing the prefix should save memory, especially in situations where prefix length >> completion length.

Some reference implementations:
- https://github.com/frankxwang/dpo-prefix-sharing
- https://github.com/li-plus/flash-preference/tree/main

### Motivation

Many classes of problems involving large contexts end up being dominated by prompt length. This should help with memory savings in those situations

### Your contribution

I would love to take a stab at a PR, but want to discuss the feasibility and interest in adding support for this in TRL

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.