intel / intel/cartwheel-ffmpeg

Runtime issues encountered with FFmpeg's VSR integration with software-based encoders

Open
#365 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
No language data
Stars
110
Forks
38
PR merge metrics
No merged PRs in 30d

Description

Hello there,

I've built Intel's VSR from the [reisr4nas branch](https://github.com/intel/cartwheel-ffmpeg/tree/raisr4nas) of ffmpeg cartwheel, and can likewise confirm that it works as expected; at least in GPU modes where OpenCL is available.

However, the same cannot be said of the out-of-the-box ASM modes with software-based encoding via `libx264` and `libx265`, as [documented on the wiki](https://github.com/OpenVisualCloud/Video-Super-Resolution-Library/tree/main#running-the-intel-library-for-vsr).

These trivial examples simply fail/exit with no output(s) on the console:

**With `libx264:**

```sh
ffmpeg -init_hw_device qsv=hw \
-init_hw_device opencl=cl@hw \
-filter_hw_device cl \
-noautorotate -i "bbb.mp4" \
-vf "raisr=filterfolder=/filters_2x/filters_highres:asm=opencl" \
-c:v libx264 -pix_fmt yuv420p -x264-params "scenecut=0" -maxrate:v 5000k -bufsize:v 1250k \
-keyint_min 120 -g:v 120 -preset:v faster -tune:v fastdecode -bf:v 0 \
-flags +global_header+cgop -f mp4 -y libx264_highres.mp4
```

**With `libx265:**

```sh
ffmpeg -init_hw_device qsv=hw \
-init_hw_device opencl=cl@hw \
-filter_hw_device cl \
-noautorotate -i "bbb.mp4" \
-vf "raisr=filterfolder=/filters_2x/filters_highres:asm=opencl" \
-c:v libx265 -pix_fmt yuv420p -x265-params "scenecut=0" -maxrate:v 5000k -bufsize:v 1250k \
-keyint_min 120 -g:v 120 -preset:v faster -tune:v fastdecode -bf:v 0 \
-flags +global_header+cgop -f mp4 -y libx265_highres.mp4
```

Now, if an OpenCL hardware context is provided as with the trivial example below for `libx264`, and `asm=opencl` is set, the filter runs/executes as expected:

```sh
ffmpeg -init_hw_device qsv=hw \
-init_hw_device opencl=cl@hw \
-filter_hw_device cl \
-noautorotate -i "bbb.mp4" \
-vf "raisr=filterfolder=/filters_2x/filters_highres:asm=opencl" \
-c:v libx264 -pix_fmt yuv420p -x264-params "scenecut=0" -maxrate:v 5000k -bufsize:v 1250k \
-keyint_min 120 -g:v 120 -preset:v faster -tune:v fastdecode -bf:v 0 \
-flags +global_header+cgop -f mp4 -y libx264_highres.mp4
```

However, attempting the same with `libx265` fails with the error:

```sh
[libx265 @ 0x556433417c80] Unknown picture type encountered.
[vost#0:0/libx265 @ 0x5564333c5d40] Error submitting video frame to the encoder
[vost#0:0/libx265 @ 0x5564333c5d40] Error encoding a frame: Generic error in an external library
[vost#0:0/libx265 @ 0x5564333c5d40] Task finished with error code: -542398533 (Generic error in an external library)
[vost#0:0/libx265 @ 0x5564333c5d40] Terminating thread with return code -542398533 (Generic error in an external library)
[out#0/mp4 @ 0x5564333fb8c0] video:0KiB audio:135KiB subtitle:0KiB other streams:0KiB global headers:3KiB muxing overhead: 1.210572%
frame= 0 fps=0.0 q=0.0 Lsize= 137KiB time=N/A bitrate=N/A dup=2 drop=0 speed=N/A
x265 [info]: frame I: 1, Avg QP:35.82 kb/s: 923.52

encoded 1 frames in 2.38s (0.42 fps), 923.52 kb/s, Avg QP:35.82
```

Regardless of the OpenCL device type/context created.

In brief: The filter is virtually unusable with the software-based `libx264` and `libx265` encoders, as-is, out of the box.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the provided FFmpeg commands from the wiki with the raisr filter, OpenCL context, and both libx264 and libx265 encoders. Compare the software-encoder behavior and error output; done means the filter completes successfully and produces usable encoded output with both encoders.

Written by the indexing model from the issue text.

Assessment

Domain
audio-video-rtc
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.