Support multi-GPU and multi-NVDEC parallel video decoding to reduce single-video latency

🚀 The feature

I would like to propose adding support for collaborative video decoding that exploits both:

1. multiple GPUs, and
2. multiple NVDEC engines within each GPU, to reduce the latency of decoding a single video.

Motivation, pitch

Video decoding is increasingly on the critical path of modern ML workloads, especially for multimodal models that process long videos. In these settings, time-to-first-frame (or time-to-first-token) is often dominated by decoding latency rather than model execution.

While GPU decoding already improves throughput, we observe that:

• When decoding a single video, typically only one NVDEC engine is used.
• Decoding is confined to a single GPU even when multiple GPUs are available.
• Parallelism is primarily video-level, rather than intra-video.

Enabling collaborative decoding across GPUs and NVDEC engines can substantially reduce single-video latency while maintaining high throughput.

We have explored this design in a research prototype and observed significant latency reductions.

For additional background and experimental results, we describe a similar design in:
Enabling Disaggregated Multi-Stage MLLM Inference via GPU-Internal Scheduling and Resource Sharing, https://arxiv.org/abs/2512.17574

If helpful, we’d be glad to submit a PR with further details.

No files, tests, or entry points are named. Start by reviewing the existing GPU video-decoding architecture and the linked research prototype, then define how work would be divided across GPUs and NVDEC engines. Done would require a concrete implementation scope and evidence that single-video latency improves.