huggingface / huggingface/trl

Packing with VLMs

Open
#4,339 5 comments 0 reactions 0 assignees View on GitHub
✨ enhancement 🏋 SFT
Dominant language
Python
Stars
19.3k
Forks
3k
Avg merge
1d 20h
Merged PRs (30d)
194

Description

### Feature request

Per https://github.com/axolotl-ai-cloud/axolotl/issues/3131, would it be possible to implement packing (in SFT) for VLMs? This is currently disabled in SFTrainer by reading the class of the provided processor.

### Motivation

This would be super helpful for reducing the memory footprint of training VLMs.

### Your contribution

If everything is processed into tokens, shouldn't packing with other modalities be straightforward (ignoring memory issues with the vision encoder)? I'm not knowledgeable in how padding_free packing interacts with flash attention 2. Is this the reason it is not straightforward?

Contributor guide

Open the contributing guide

Research direction

Start by tracing how SFTrainer reads the provided processor class to disable SFT packing, then inspect how padding_free packing interacts with multimodal inputs and flash attention 2. The work is done when VLM SFT supports packing without breaking vision modalities or the stated attention path, with coverage for the relevant training behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.