NVIDIA-NeMo / NVIDIA-NeMo/Automodel
[VLM] Support pre-extracted video frames (list of image paths) in _preload_media
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 963
- Forks
- 318
- Avg merge
- 3d 20h
- Merged PRs (30d)
- 143
Description
Problem
_preload_media() in nemo_automodel/components/datasets/vlm/utils.py only handles videos given as a file path string, decoding them with decord:
elif media_type == "video":
vid = item.get("video")
if isinstance(vid, str):
... # decord decode path
# <-- no branch for pre-extracted frame lists; the item falls through silently
When a dataset sample provides a video as a list of pre-extracted frame image paths -- a common format for large-scale video training where frames are dumped offline (also accepted by LLaMA-Factory's videos field, qwen_vl_utils, etc.):
{"type": "video", "video": ["frames/vid001/0000.jpg", "frames/vid001/0001.jpg", "..."], "fps": 2.0}
the item falls through silently. item["video"] remains a list of strings and is passed as-is to the HF processor (videos=[...], do_sample_frames=False) by the conversation-based datasets (PreTokenizedDatasetWrapper / RobustDatasetWrapper in nemo_automodel/components/datasets/vlm/datasets.py). The processor's video backend (PyAV / torchcodec / torchvision) then tries to open each JPEG/PNG as a video container and fails (e.g. RuntimeError: Failed to load video ...). It also forces the optional video-decoder stack (nemo-automodel[vlm-media]) to be installed even though no video decoding is actually needed.
Motivation
- Pre-extracting frames offline is standard practice for large-scale video SFT: decode once, store JPEGs, reuse across epochs/ablations -- much cheaper than decoding mp4s in every dataloader worker, and avoids decoder flakiness.
- Removes the hard dependency on decord/PyAV in minimal or container environments where the media stack is not installed.
- Datasets produced for other frameworks (e.g. LLaMA-Factory) are often already in this shape, so supporting it lowers the migration barrier.
Proposed behavior
In _preload_media, accept video as a list/tuple of image path strings:
- Load each entry directly with
Image.open(...).convert("RGB")-- no video decoder involved. - Pad the frame list to
temporal_patch_sizealignment by repeating the last frame (mirroring the existing decord path). - When
preserve_video_metadata=True, read an optional per-samplefpsfield from the video item (falling back toprocessor.video_processor.fps) and set_frame_indices = list(range(len(frames))), so_build_video_metadata()produces correctVideoMetadataand the processor inserts correct timestamps.
Reference implementation (tested locally with Qwen3-VL-30B-A3B SFT):
elif isinstance(vid, (list, tuple)) and vid and all(isinstance(f, str) for f in vid):
# Pre-extracted frame sequence: every entry is an image path,
# so frames load directly -- no video decoder required.
temporal_patch_size = 2
if processor is not None and hasattr(processor, "video_processor"):
temporal_patch_size = getattr(processor.video_processor, "temporal_patch_size", 2)
frames = [Image.open(f).convert("RGB") for f in vid]
indices = list(range(len(frames)))
# Pad to temporal_patch_size alignment by repeating the last
# frame, mirroring the decord path above.
remainder = len(indices) % temporal_patch_size
if remainder != 0:
pad = temporal_patch_size - remainder
frames.extend([frames[-1]] * pad)
indices.extend([indices[-1]] * pad)
item["video"] = frames
if preserve_video_metadata:
fps = item.get("fps")
if fps is None and processor is not None and hasattr(processor, "video_processor"):
fps = getattr(processor.video_processor, "fps", None)
item["_video_fps"] = fps
item["_frame_indices"] = indices
Happy to open a PR with this change plus a unit test if the maintainers agree with the direction.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in nemo_automodel/components/datasets/vlm/utils.py at _preload_media, then inspect how PreTokenizedDatasetWrapper and RobustDatasetWrapper use the result in nemo_automodel/components/datasets/vlm/datasets.py. Add focused coverage for lists and tuples of frame paths, including temporal padding and optional metadata, and verify that the frames reach the processor as RGB images without video decoding.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, machine-learning
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100