meta-pytorch / meta-pytorch/torchcodec
Add motion‑vector extraction API (MV‑only mode + frame types) for ML workflows
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 125
- Avg merge
- 22h 47m
- Merged PRs (30d)
- 54
Description
🚀 The feature
The idea is to expose compressed-domain motion vectors from VideoDecoder.
- Add an API to fetch per‑frame motion vectors returned by FFmpeg via
AV_FRAME_DATA_MOTION_VECTORS. - Support MV‑only mode that skips
RGBdecoding for higher throughput. - Optionally return frame type (
I/P/B) alongside vectors. NVDEC: if motion vectors are available via theCUDA/NVDECpath, consider parity with the CPU API; otherwise it can be CPU‑only initially.
Possible API shapes:
VideoDecoder.get_motion_vectors_at(indices=...) -> MotionVectorBatchVideoDecoder.get_frames_at(..., with_motion_vectors=True)VideoDecoder(..., decode_frames=False, return_motion_vectors=True)
Motion vector format could mirror AVMotionVector fields (N x 10):
source, w, h, src_x, src_y, dst_x, dst_y, motion_x, motion_y, motion_scale as int32 on CPU.
Motivation, pitch
TorchCodec is a great fit for video ML pipelines, but it currently returns only decoded frames. Many models can benefit from compressed‑domain motion vectors for fast motion cues, tracking, or flow warm‑starts.
Exposing AV_FRAME_DATA_MOTION_VECTORS (and an MV‑only fast path) would enable high‑throughput training/inference without full optical flow.
Related work:
https://arxiv.org/html/2510.17427v1
Reference implementation:
https://github.com/LukasBommes/mv-extractor
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the VideoDecoder entry point and trace how FFmpeg exposes AV_FRAME_DATA_MOTION_VECTORS, then compare the CPU and NVDEC paths. Resolve the API shape, MV-only behavior, frame-type output, and motion-vector format before implementation. Done means the chosen API supports the documented workflow and has coverage for the supported decoding path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- audio-video-rtc, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100