Video inputs for multimodal LLM
- Dominant language
- Python
- Stars
- 19.3k
- Forks
- 3k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 194
Description
### Feature request
Hello,
currently, it is not possible to train a multimodal LLM with video input (e.g., Qwen3-VL) where the video input is handled separately from image inputs. At the moment, only the "images" key can be used as model input in the dataset, and it is processed slightly differently.
It is necessary to pass a list of images representing the video to the processor via processor(videos=videos, ...), instead of the current approach using only processor(images=images, ...).
It would be beneficial to extend the default dataset keys to include prompt, image/images, and video. Additionally, it would be useful to add support for setting do_sample_frames=False in the processor, allowing control over whether the video is frame-sampled or processed in full.
### Motivation
Currently, it is not possible to train an LLM such as Qwen3-VL with a video input that is not treated as a set of separate images at the input.
### Your contribution
I would be happy to collaborate on this feature request by creating a potential PR to extend the existing implementation of GRPOTrainer.
Contributor guide
Assessment
This issue has not been assessed yet.