BotBlake / BotBlake/jellybench_py
Crash if ffmpeg does not report speed or maxrss
- Dominant language
- Python
- Stars
- 28
- Forks
- 12
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 1
Description
### Classifications
- [x] Running `python ./jellybench_py/hwi.py` manually (within the Poetry shell) works and produces correct results.
- [x] This happens on a Virtual Hardware (VM, WSL, ...)
- [x] The issue occurs during the benchmark section of the script (after `Starting Benchmark...`).
- [ ] A server issue or faulty server response is causing the problem.
- [ ] jellybench was run in `--debug` mode (recommended for issue reports).
- [x] I am using the latest version (branch `develop`) of jellybench. I understand older versions are not supported.
### Description
When running jellybench in normal mode (non-debug), the script crashes during part of the progress bar rendering step. If ffmpeg does not output a "speed" or "maxrss", python raises a `KeyError`. This does not happen in debug mode as the progress bar isn't rendered.
### Error Code or Output
```shell
Traceback (most recent call last):
File "/usr/local/bin/jellybench", line 8, in
sys.exit(main())
^^^^^^
File "/usr/local/lib/python3.12/dist-packages/jellybench_py/core.py", line 948, in main
return cli()
^^^^^
File "/usr/local/lib/python3.12/dist-packages/jellybench_py/core.py", line 910, in cli
valid, runs, result = benchmark(
^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/jellybench_py/core.py", line 357, in benchmark
"single_worker_speed": max_pass_run_data["speed"],
~~~~~~~~~~~~~~~~~^^^^^^^^^
KeyError: 'speed'
OR
Traceback (most recent call last):
File "/usr/local/bin/jellybench", line 8, in
sys.exit(main())
^^^^^^
File "/usr/local/lib/python3.12/dist-packages/jellybench_py/core.py", line 948, in main
return cli()
^^^^^
File "/usr/local/lib/python3.12/dist-packages/jellybench_py/core.py", line 910, in cli
valid, runs, result = benchmark(
^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/jellybench_py/core.py", line 358, in benchmark
"single_worker_rss_kb": max_pass_run_data["rss_kb"],
~~~~~~~~~~~~~~~~~^^^^^^^^^^
KeyError: 'rss_kb'
```
### Steps to Reproduce
1. Start jellybench normally on a system where ffmpeg fails in a specific manner (see additional context)
2. ffmpeg fails and does not print `speed` or `maxrss`
3. Observe traceback
### Operating System
Linux
### System Architecture
x86_64 (64-bit)
### Python Version
3.12.3
### CPU[s]
AMD Ryzen 7 5700G with Radeon Graphics
### GPU[s]
Cezanne [Radeon Vega Series / Radeon Vega Mobile Series]
### Additional Context
I am running this in docker, according to my discussion [here](https://github.com/BotBlake/jellybench_py/discussions/85)
Working ffmpeg benchmark
```console
root@0d182ef0c537:/usr/src/jellybench_py# /usr/src/jellybench_py/jellybench_data/ffmpeg/ffmpeg_files/ffmpeg -init_hw_device vaapi=va:/dev/dri/by-path/pci-0000:0d:00.0-render -hwaccel vaapi -hwaccel_output_format vaapi -c:v hevc -i /usr/src/jellybench_py/jellybench_data/videos/Test%20Jellyfin%204K%20HEVC%2010bit%20150M.mp4 -noautoscale -an -sn -vf "scale_vaapi=-1:528:format=nv12" -c:v hevc_vaapi -b:v 79616000 -maxrate 79616000 -f null - -benchmark
ffmpeg version 7.0.2-Jellyfin Copyright (c) 2000-2024 the FFmpeg developers
built with gcc 13 (Ubuntu 13.3.0-6ubuntu2~24.04)
configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto=auto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-opencl --enable-libdrm --enable-libxml2 --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libharfbuzz --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libsvtav1 --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
libavutil 59. 8.100 / 59. 8.100
libavcodec 61. 3.100 / 61. 3.100
libavformat 61. 1.100 / 61. 1.100
libavdevice 61. 1.100 / 61. 1.100
libavfilter 10. 1.100 / 10. 1.100
libswscale 8. 1.100 / 8. 1.100
libswresample 5. 1.100 / 5. 1.100
libpostproc 58. 1.100 / 58. 1.100
amdgpu: os_same_file_description couldn't determine if two DRM fds reference the same file description.
If they do, bad things may happen!
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/usr/src/jellybench_py/jellybench_data/videos/Test%20Jellyfin%204K%20HEVC%2010bit%20150M.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2mp41
encoder : Lavf61.1.100
Duration: 00:00:29.95, start: 0.000000, bitrate: 147664 kb/s
Stream #0:0[0x1](und): Video: hevc (Main 10) (hvc1 / 0x31637668), yuv420p10le(tv, bt709/bt709/unknown, progressive), 3840x2160 [SAR 1:1 DAR 16:9], 147661 kb/s, 60 fps, 60 tbr, 15360 tbn (default)
Metadata:
handler_name : VideoHandler
vendor_id : [0][0][0][0]
encoder : Lavc61.3.100 hevc_videotoolbox
Stream mapping:
Stream #0:0 -> #0:0 (hevc (native) -> hevc (hevc_vaapi))
Press [q] to stop, [?] for help
[hevc_vaapi @ 0x5581a1a4e9c0] Driver does not support some wanted packed headers (wanted 0xd, found 0x1).
Output #0, null, to 'pipe:':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2mp41
encoder : Lavf61.1.100
Stream #0:0(und): Video: hevc (Main), vaapi(tv, bt709, progressive), 939x528 [SAR 2816:2817 DAR 16:9], q=2-31, 79616 kb/s, 60 fps, 60 tbn (default)
Metadata:
handler_name : VideoHandler
vendor_id : [0][0][0][0]
encoder : Lavc61.3.100 hevc_vaapi
[out#0/null @ 0x5581a1a4a740] video:288661KiB audio:0KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: unknown
frame= 1797 fps=127 q=-0.0 Lsize=N/A time=00:00:29.95 bitrate=N/A speed=2.11x
bench: utime=1.247s stime=0.995s rtime=14.199s
bench: maxrss=530632KiB
```
Broken ffmpeg benchmark
```console
root@0d182ef0c537:/usr/src/jellybench_py# /usr/src/jellybench_py/jellybench_data/ffmpeg/ffmpeg_files/ffmpeg -c:v hevc -i /usr/src/jellybench_py/jellybench_data/videos/Test%20Jellyfin%204K%20HEVC%2010bit%20150M.mp4 -noautoscale -an -sn -vf "scale=trunc(min(max(iw\,ih*a)\,528)/2)*2:trunc(ow/a/2)*2,format=yuv420p" -c:v libx265 -preset veryfast -b:v 79616000 -maxrate 79616000 -f null - -benchmark
ffmpeg version 7.0.2-Jellyfin Copyright (c) 2000-2024 the FFmpeg developers
built with gcc 13 (Ubuntu 13.3.0-6ubuntu2~24.04)
configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto=auto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-opencl --enable-libdrm --enable-libxml2 --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libharfbuzz --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libsvtav1 --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
libavutil 59. 8.100 / 59. 8.100
libavcodec 61. 3.100 / 61. 3.100
libavformat 61. 1.100 / 61. 1.100
libavdevice 61. 1.100 / 61. 1.100
libavfilter 10. 1.100 / 10. 1.100
libswscale 8. 1.100 / 8. 1.100
libswresample 5. 1.100 / 5. 1.100
libpostproc 58. 1.100 / 58. 1.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/usr/src/jellybench_py/jellybench_data/videos/Test%20Jellyfin%204K%20HEVC%2010bit%20150M.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2mp41
encoder : Lavf61.1.100
Duration: 00:00:29.95, start: 0.000000, bitrate: 147664 kb/s
Stream #0:0[0x1](und): Video: hevc (Main 10) (hvc1 / 0x31637668), yuv420p10le(tv, bt709/bt709/unknown, progressive), 3840x2160 [SAR 1:1 DAR 16:9], 147661 kb/s, 60 fps, 60 tbr, 15360 tbn (default)
Metadata:
handler_name : VideoHandler
vendor_id : [0][0][0][0]
encoder : Lavc61.3.100 hevc_videotoolbox
Stream mapping:
Stream #0:0 -> #0:0 (hevc (native) -> hevc (libx265))
Press [q] to stop, [?] for help
x265 [info]: HEVC encoder version 3.5+1-f0c1022b6
x265 [info]: build info [Linux][GCC 13.2.0][64 bit] 8bit+10bit+12bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
x265 [info]: Main profile, Level-5 (High tier)
x265 [info]: Thread pool created using 16 threads
x265 [info]: Slices : 1
x265 [info]: frame threads / pool features : 4 / wpp(5 rows)
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
x265 [warning]: Source height < 720p; disabling lookahead-slices
x265 [warning]: VBV maxrate specified, but no bufsize, ignored
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
x265 [info]: Coding QT: max CU size, min CU size : 64 / 8
x265 [info]: Residual QT: max TU size, max depth : 32 / 1 inter / 1 intra
x265 [info]: ME / range / subpel / merge : hex / 57 / 1 / 2
x265 [info]: Keyframe min / max / scenecut / bias : 25 / 250 / 40 / 5.00
x265 [info]: Lookahead / bframes / badapt : 15 / 4 / 0
x265 [info]: b-pyramid / weightp / weightb : 1 / 1 / 0
x265 [info]: References / ref-limit cu / depth : 2 / on / on
x265 [info]: AQ: mode / str / qg-size / cu-tree : 2 / 1.0 / 32 / 1
x265 [info]: Rate Control / qCompress : ABR-79616 kbps / 0.60
x265 [info]: tools: rd=2 psy-rd=2.00 early-skip rskip mode=1 signhide tmvp
x265 [info]: tools: fast-intra strong-intra-smoothing deblock sao
Output #0, null, to 'pipe:':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2mp41
encoder : Lavf61.1.100
Stream #0:0(und): Video: hevc, yuv420p(tv, bt709/bt709/unknown, progressive), 528x296 [SAR 296:297 DAR 16:9], q=2-31, 79616 kb/s, 60 fps, 60 tbn (default)
Metadata:
handler_name : VideoHandler
vendor_id : [0][0][0][0]
encoder : Lavc61.3.100 libx265
Side data:
cpb: bitrate max/min/avg: 79616000/0/79616000 buffer size: 0 vbv_delay: N/A
[out#0/null @ 0x55d1559e4200] video:69831KiB audio:0KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: unknown
frame= 1797 fps= 59 q=0.3 Lsize=N/A time=00:00:29.91 bitrate=N/A speed=0.988x
bench: utime=180.647s stime=2.023s rtime=30.286s
bench: maxrss=1003308KiB
x265 [info]: frame I: 8, Avg QP:0.00 kb/s: 37592.58
x265 [info]: frame P: 353, Avg QP:0.00 kb/s: 27251.53
x265 [info]: frame B: 1436, Avg QP:1.18 kb/s: 16984.90
x265 [info]: Weighted P-Frames: Y:0.0% UV:0.0%
x265 [info]: consecutive B-frames: 0.6% 0.0% 0.0% 0.0% 99.4%
encoded 1797 frames in 30.13s (59.64 fps), 19093.40 kb/s, Avg QP:0.94
```
Note the missing `speed` and `maxrss` output. I have not looked into _why_ they are missing, I just wanted to get the python script to better handle failures. As this is happening during the progress bar steps, I don't think it's all that critical to the output of the script.
### Possible Solution
Minor code changes; I will open a PR with the fixes I used to get the tool to finish working.
Contributor guide
Research direction
Start in jellybench_py/core.py at benchmark(), especially the result construction around the reported accesses to max_pass_run_data["speed"] and ["rss_kb"]. Reproduce the normal, non-debug run with an ffmpeg benchmark that omits those fields, then verify the benchmark no longer crashes when either value is absent.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli, performance
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100