NVIDIA-NeMo / NVIDIA-NeMo/Automodel

[VLM] Support pre-extracted video frames (list of image paths) in _preload_media

Open
#3,168 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

community-request enhancement waiting-on-customer
Dominant language
Python
Stars
963
Forks
318
Avg merge
3d 20h
Merged PRs (30d)
143

Description

Problem

_preload_media() in nemo_automodel/components/datasets/vlm/utils.py only handles videos given as a file path string, decoding them with decord:

https://github.com/NVIDIA-NeMo/Automodel/blob/main/nemo_automodel/components/datasets/vlm/utils.py#L186-L204

elif media_type == "video":
    vid = item.get("video")
    if isinstance(vid, str):
        ...  # decord decode path
    # <-- no branch for pre-extracted frame lists; the item falls through silently

When a dataset sample provides a video as a list of pre-extracted frame image paths -- a common format for large-scale video training where frames are dumped offline (also accepted by LLaMA-Factory's videos field, qwen_vl_utils, etc.):

{"type": "video", "video": ["frames/vid001/0000.jpg", "frames/vid001/0001.jpg", "..."], "fps": 2.0}

the item falls through silently. item["video"] remains a list of strings and is passed as-is to the HF processor (videos=[...], do_sample_frames=False) by the conversation-based datasets (PreTokenizedDatasetWrapper / RobustDatasetWrapper in nemo_automodel/components/datasets/vlm/datasets.py). The processor's video backend (PyAV / torchcodec / torchvision) then tries to open each JPEG/PNG as a video container and fails (e.g. RuntimeError: Failed to load video ...). It also forces the optional video-decoder stack (nemo-automodel[vlm-media]) to be installed even though no video decoding is actually needed.

Motivation

  • Pre-extracting frames offline is standard practice for large-scale video SFT: decode once, store JPEGs, reuse across epochs/ablations -- much cheaper than decoding mp4s in every dataloader worker, and avoids decoder flakiness.
  • Removes the hard dependency on decord/PyAV in minimal or container environments where the media stack is not installed.
  • Datasets produced for other frameworks (e.g. LLaMA-Factory) are often already in this shape, so supporting it lowers the migration barrier.

Proposed behavior

In _preload_media, accept video as a list/tuple of image path strings:

  • Load each entry directly with Image.open(...).convert("RGB") -- no video decoder involved.
  • Pad the frame list to temporal_patch_size alignment by repeating the last frame (mirroring the existing decord path).
  • When preserve_video_metadata=True, read an optional per-sample fps field from the video item (falling back to processor.video_processor.fps) and set _frame_indices = list(range(len(frames))), so _build_video_metadata() produces correct VideoMetadata and the processor inserts correct timestamps.

Reference implementation (tested locally with Qwen3-VL-30B-A3B SFT):

elif isinstance(vid, (list, tuple)) and vid and all(isinstance(f, str) for f in vid):
    # Pre-extracted frame sequence: every entry is an image path,
    # so frames load directly -- no video decoder required.
    temporal_patch_size = 2
    if processor is not None and hasattr(processor, "video_processor"):
        temporal_patch_size = getattr(processor.video_processor, "temporal_patch_size", 2)
    frames = [Image.open(f).convert("RGB") for f in vid]
    indices = list(range(len(frames)))
    # Pad to temporal_patch_size alignment by repeating the last
    # frame, mirroring the decord path above.
    remainder = len(indices) % temporal_patch_size
    if remainder != 0:
        pad = temporal_patch_size - remainder
        frames.extend([frames[-1]] * pad)
        indices.extend([indices[-1]] * pad)
    item["video"] = frames
    if preserve_video_metadata:
        fps = item.get("fps")
        if fps is None and processor is not None and hasattr(processor, "video_processor"):
            fps = getattr(processor.video_processor, "fps", None)
        item["_video_fps"] = fps
        item["_frame_indices"] = indices

Happy to open a PR with this change plus a unit test if the maintainers agree with the direction.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in nemo_automodel/components/datasets/vlm/utils.py at _preload_media, then inspect how PreTokenizedDatasetWrapper and RobustDatasetWrapper use the result in nemo_automodel/components/datasets/vlm/datasets.py. Add focused coverage for lists and tuples of frame paths, including temporal padding and optional metadata, and verify that the frames reach the processor as RGB images without video decoding.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, machine-learning
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.