mpv-player / mpv-player/mpv

Shader stage expansion for temporal shaders/algorithms?

Open
#8,137 20 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

core:video meta:feature-request
Dominant language
C
Stars
37k
Forks
3.5k
Avg merge
1d 10h
Merged PRs (30d)
22

Description

Hello, I have been wondering if it was possible to expand the mpv video output shader stage with the following features?
I know this might not be on the devs' main priority, but if I were to try to implement them, where should I start?

These are ordered by immediate usefulness.

  1. Expose the MPEG motion vectors as a source plane. (eg. a new MOTION source hook instead of just LUMA and CHROMA)
  2. Allowing a shader to read previous or future source hooks. (eg. accessing the LUMA of the current frame and the previous frame)
  3. Allow textures to stay alive across different frames. (eg. accessing a texture saved from the previous video frames)
  4. Allow writing to 3D textures. (in the form of a virtual 3D hook?)

(1) can allow the development more complex shaders which take in account motion.
(1, 2) can allow the implementation of MISR algorithms (multiple image super resolution) for video.
(1, 2, 3) can allow the implementation of temporal algorithms (eg. Unity's TSS) and RNNs.
(1, 2, 3, 4) can allow the implementation of very efficient Convolutional RNNs.

Furthermore, by implementing (2) alone and allowing people or the shader to choose a target framerate can allow the implementation of custom temporal interpolation algorithms. (--tscale) This can be useful for implementing frame interpolation algorithms such as DAIN.

mpv is already a great player thanks to its open shader stage. I believe adding these features will make mpv unique and bridge the gap between algorithms developed in academia and algorithms used in practice.

Thanks in advance.

Edit: I meant (4) writing to 3D textures. I know that mpv already supports read only 3D textures.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files, tests, or entry points are named. Start by locating mpv's video output shader-stage implementation and its existing source-hook and texture handling, then determine which of the four requested capabilities can be scoped independently. Done would require an agreed design, implementation boundaries, and tests for the selected feature.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
audio-video-rtc, computer-graphics
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.