huggingface / huggingface/trl

Video inputs for multimodal LLM

Open
#5,373 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
19.3k
Forks
3k
Avg merge
1d 20h
Merged PRs (30d)
194

Description

### Feature request

Hello,

currently, it is not possible to train a multimodal LLM with video input (e.g., Qwen3-VL) where the video input is handled separately from image inputs. At the moment, only the "images" key can be used as model input in the dataset, and it is processed slightly differently.

It is necessary to pass a list of images representing the video to the processor via processor(videos=videos, ...), instead of the current approach using only processor(images=images, ...).

It would be beneficial to extend the default dataset keys to include prompt, image/images, and video. Additionally, it would be useful to add support for setting do_sample_frames=False in the processor, allowing control over whether the video is frame-sampled or processed in full.

### Motivation

Currently, it is not possible to train an LLM such as Qwen3-VL with a video input that is not treated as a set of separate images at the input.

### Your contribution

I would be happy to collaborate on this feature request by creating a potential PR to extend the existing implementation of GRPOTrainer.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.