intel / intel/cartwheel-ffmpeg
Runtime issues encountered with FFmpeg's VSR integration with software-based encoders
- Dominant language
- No language data
- Stars
- 110
- Forks
- 38
- PR merge metrics
- No merged PRs in 30d
Description
Hello there,
I've built Intel's VSR from the [reisr4nas branch](https://github.com/intel/cartwheel-ffmpeg/tree/raisr4nas) of ffmpeg cartwheel, and can likewise confirm that it works as expected; at least in GPU modes where OpenCL is available.
However, the same cannot be said of the out-of-the-box ASM modes with software-based encoding via `libx264` and `libx265`, as [documented on the wiki](https://github.com/OpenVisualCloud/Video-Super-Resolution-Library/tree/main#running-the-intel-library-for-vsr).
These trivial examples simply fail/exit with no output(s) on the console:
**With `libx264:**
```sh
ffmpeg -init_hw_device qsv=hw \
-init_hw_device opencl=cl@hw \
-filter_hw_device cl \
-noautorotate -i "bbb.mp4" \
-vf "raisr=filterfolder=/filters_2x/filters_highres:asm=opencl" \
-c:v libx264 -pix_fmt yuv420p -x264-params "scenecut=0" -maxrate:v 5000k -bufsize:v 1250k \
-keyint_min 120 -g:v 120 -preset:v faster -tune:v fastdecode -bf:v 0 \
-flags +global_header+cgop -f mp4 -y libx264_highres.mp4
```
**With `libx265:**
```sh
ffmpeg -init_hw_device qsv=hw \
-init_hw_device opencl=cl@hw \
-filter_hw_device cl \
-noautorotate -i "bbb.mp4" \
-vf "raisr=filterfolder=/filters_2x/filters_highres:asm=opencl" \
-c:v libx265 -pix_fmt yuv420p -x265-params "scenecut=0" -maxrate:v 5000k -bufsize:v 1250k \
-keyint_min 120 -g:v 120 -preset:v faster -tune:v fastdecode -bf:v 0 \
-flags +global_header+cgop -f mp4 -y libx265_highres.mp4
```
Now, if an OpenCL hardware context is provided as with the trivial example below for `libx264`, and `asm=opencl` is set, the filter runs/executes as expected:
```sh
ffmpeg -init_hw_device qsv=hw \
-init_hw_device opencl=cl@hw \
-filter_hw_device cl \
-noautorotate -i "bbb.mp4" \
-vf "raisr=filterfolder=/filters_2x/filters_highres:asm=opencl" \
-c:v libx264 -pix_fmt yuv420p -x264-params "scenecut=0" -maxrate:v 5000k -bufsize:v 1250k \
-keyint_min 120 -g:v 120 -preset:v faster -tune:v fastdecode -bf:v 0 \
-flags +global_header+cgop -f mp4 -y libx264_highres.mp4
```
However, attempting the same with `libx265` fails with the error:
```sh
[libx265 @ 0x556433417c80] Unknown picture type encountered.
[vost#0:0/libx265 @ 0x5564333c5d40] Error submitting video frame to the encoder
[vost#0:0/libx265 @ 0x5564333c5d40] Error encoding a frame: Generic error in an external library
[vost#0:0/libx265 @ 0x5564333c5d40] Task finished with error code: -542398533 (Generic error in an external library)
[vost#0:0/libx265 @ 0x5564333c5d40] Terminating thread with return code -542398533 (Generic error in an external library)
[out#0/mp4 @ 0x5564333fb8c0] video:0KiB audio:135KiB subtitle:0KiB other streams:0KiB global headers:3KiB muxing overhead: 1.210572%
frame= 0 fps=0.0 q=0.0 Lsize= 137KiB time=N/A bitrate=N/A dup=2 drop=0 speed=N/A
x265 [info]: frame I: 1, Avg QP:35.82 kb/s: 923.52
encoded 1 frames in 2.38s (0.42 fps), 923.52 kb/s, Avg QP:35.82
```
Regardless of the OpenCL device type/context created.
In brief: The filter is virtually unusable with the software-based `libx264` and `libx265` encoders, as-is, out of the box.
Contributor guide
Research direction
Start by reproducing the provided FFmpeg commands from the wiki with the raisr filter, OpenCL context, and both libx264 and libx265 encoders. Compare the software-encoder behavior and error output; done means the filter completes successfully and produces usable encoded output with both encoders.
Written by the indexing model from the issue text.
Assessment
- Domain
- audio-video-rtc
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100