BotBlake / BotBlake/jellybench_py
Crash is run with 1 worker reports speed <1
- Dominant language
- Python
- Stars
- 28
- Forks
- 12
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 1
Description
### Classifications
- [x] Running `python ./jellybench_py/hwi.py` manually (within the Poetry shell) works and produces correct results.
- [ ] This happens on a Virtual Hardware (VM, WSL, ...)
- [x] The issue occurs during the benchmark section of the script (after `Starting Benchmark...`).
- [ ] A server issue or faulty server response is causing the problem.
- [x] jellybench was run in `--debug` mode (recommended for issue reports).
- [x] I am using the latest version (branch `develop`) of jellybench. I understand older versions are not supported.
### Description
Hi,
I'm testing Jellybench on various systems. If the system can't handle a run with a speed of >= 1 Jellybench crashes with the following error:
### Error Code or Output
```shell
> > > > Failed: ['performance']
| > > Current Test: 2160p - 720p
| > > > Current Device: cpu
| > > > ffmpeg command: ./jellybench_data/ffmpeg/ffmpeg_files/ffmpeg -c:v h264 -i /home/redacted/jellybench_py/jellybench_data/videos/Test%20Jellyfin%204K%20AVC%20150M.mp4 -noautoscale -an -sn -vf "scale=trunc(min(max(iw\,ih*a)\,528)/2)*2:trunc(ow/a/2)*2,format=yuv420p" -c:v libx265 -preset veryfast -b:v 3616000 -maxrate 3616000 -f null - -benchmark
| > > > > Starting run with 1 Workers... completed with speed 0.99
| > > > > Failed: ['limited']
Traceback (most recent call last):
File "/home/redacted/jellybench_py/bin/jellybench", line 6, in
sys.exit(main())
^^^^^^
File "/home/redacted/jellybench_py/jellybench_py/core.py", line 1045, in main
return cli()
^^^^^
File "/home/redacted/jellybench_py/jellybench_py/core.py", line 998, in cli
valid, runs, result = benchmark(
^^^^^^^^^^
File "/home/redacted/jellybench_py/jellybench_py/core.py", line 360, in benchmark
"single_worker_speed": max_pass_run_data["speed"],
~~~~~~~~~~~~~~~~~^^^^^^^^^
KeyError: 'speed'
```
### Steps to Reproduce
Run Jellybench on a system that can't handle cpu intensive transcoding tasks
### Operating System
Linux
### System Architecture
x86_64 (64-bit)
### Python Version
3.12.3
### CPU[s]
Intel(R) Core(TM) i5-7500T CPU @ 2.70GHz
### GPU[s]
Intel HD Graphics 630
### Additional Context
_No response_
### Possible Solution
Hotfix is to set "speed" and "rss_kb"to 0 if not set. Don't know how the API will handle this, though. Change line 360 and 361 in core.py from
```
"single_worker_speed": max_pass_run_data["speed"],
"single_worker_rss_kb": max_pass_run_data["rss_kb"],
```
to
```
"single_worker_speed": max_pass_run_data.get("speed", 0),
"single_worker_rss_kb": max_pass_run_data.get("rss_kb", 0),
```
Contributor guide
Assessment
This issue has not been assessed yet.