Shader stage expansion for temporal shaders/algorithms?
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 37k
- Forks
- 3.5k
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 22
Description
Hello, I have been wondering if it was possible to expand the mpv video output shader stage with the following features?
I know this might not be on the devs' main priority, but if I were to try to implement them, where should I start?
These are ordered by immediate usefulness.
- Expose the MPEG motion vectors as a source plane. (eg. a new MOTION source hook instead of just LUMA and CHROMA)
- Allowing a shader to read previous or future source hooks. (eg. accessing the LUMA of the current frame and the previous frame)
- Allow textures to stay alive across different frames. (eg. accessing a texture saved from the previous video frames)
- Allow writing to 3D textures. (in the form of a virtual 3D hook?)
(1) can allow the development more complex shaders which take in account motion.
(1, 2) can allow the implementation of MISR algorithms (multiple image super resolution) for video.
(1, 2, 3) can allow the implementation of temporal algorithms (eg. Unity's TSS) and RNNs.
(1, 2, 3, 4) can allow the implementation of very efficient Convolutional RNNs.
Furthermore, by implementing (2) alone and allowing people or the shader to choose a target framerate can allow the implementation of custom temporal interpolation algorithms. (--tscale) This can be useful for implementing frame interpolation algorithms such as DAIN.
mpv is already a great player thanks to its open shader stage. I believe adding these features will make mpv unique and bridge the gap between algorithms developed in academia and algorithms used in practice.
Thanks in advance.
Edit: I meant (4) writing to 3D textures. I know that mpv already supports read only 3D textures.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named. Start by locating mpv's video output shader-stage implementation and its existing source-hook and texture handling, then determine which of the four requested capabilities can be scoped independently. Done would require an agreed design, implementation boundaries, and tests for the selected feature.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c
- Domain
- audio-video-rtc, computer-graphics
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100