meta-pytorch / meta-pytorch/torchcodec

Add motion‑vector extraction API (MV‑only mode + frame types) for ML workflows

Open
#1,218 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.2k
Forks
125
Avg merge
22h 47m
Merged PRs (30d)
54

Description

🚀 The feature

The idea is to expose compressed-domain motion vectors from VideoDecoder.

  • Add an API to fetch per‑frame motion vectors returned by FFmpeg via AV_FRAME_DATA_MOTION_VECTORS.
  • Support MV‑only mode that skips RGB decoding for higher throughput.
  • Optionally return frame type (I/P/B) alongside vectors.
  • NVDEC: if motion vectors are available via the CUDA/NVDEC path, consider parity with the CPU API; otherwise it can be CPU‑only initially.

Possible API shapes:

  • VideoDecoder.get_motion_vectors_at(indices=...) -> MotionVectorBatch
  • VideoDecoder.get_frames_at(..., with_motion_vectors=True)
  • VideoDecoder(..., decode_frames=False, return_motion_vectors=True)

Motion vector format could mirror AVMotionVector fields (N x 10):
source, w, h, src_x, src_y, dst_x, dst_y, motion_x, motion_y, motion_scale as int32 on CPU.

Motivation, pitch

TorchCodec is a great fit for video ML pipelines, but it currently returns only decoded frames. Many models can benefit from compressed‑domain motion vectors for fast motion cues, tracking, or flow warm‑starts.
Exposing AV_FRAME_DATA_MOTION_VECTORS (and an MV‑only fast path) would enable high‑throughput training/inference without full optical flow.

Related work:
https://arxiv.org/html/2510.17427v1

Reference implementation:
https://github.com/LukasBommes/mv-extractor

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the VideoDecoder entry point and trace how FFmpeg exposes AV_FRAME_DATA_MOTION_VECTORS, then compare the CPU and NVDEC paths. Resolve the API shape, MV-only behavior, frame-type output, and motion-vector format before implementation. Done means the chosen API supports the documented workflow and has coverage for the supported decoding path.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
audio-video-rtc, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.