Packing with VLMs
- Dominant language
- Python
- Stars
- 19.3k
- Forks
- 3k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 194
Description
### Feature request
Per https://github.com/axolotl-ai-cloud/axolotl/issues/3131, would it be possible to implement packing (in SFT) for VLMs? This is currently disabled in SFTrainer by reading the class of the provided processor.
### Motivation
This would be super helpful for reducing the memory footprint of training VLMs.
### Your contribution
If everything is processed into tokens, shouldn't packing with other modalities be straightforward (ignoring memory issues with the vision encoder)? I'm not knowledgeable in how padding_free packing interacts with flash attention 2. Is this the reason it is not straightforward?
Contributor guide
Research direction
Start by tracing how SFTrainer reads the provided processor class to disable SFT packing, then inspect how padding_free packing interacts with multimodal inputs and flash attention 2. The work is done when VLM SFT supports packing without breaking vision modalities or the stated attention path, with coverage for the relevant training behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100