BotBlake / BotBlake/jellybench_py

Crash is run with 1 worker reports speed <1

Open
#102 2 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Python
Stars
28
Forks
12
Avg merge
3d 9h
Merged PRs (30d)
1

Description

### Classifications

- [x] Running `python ./jellybench_py/hwi.py` manually (within the Poetry shell) works and produces correct results.
- [ ] This happens on a Virtual Hardware (VM, WSL, ...)
- [x] The issue occurs during the benchmark section of the script (after `Starting Benchmark...`).
- [ ] A server issue or faulty server response is causing the problem.
- [x] jellybench was run in `--debug` mode (recommended for issue reports).
- [x] I am using the latest version (branch `develop`) of jellybench. I understand older versions are not supported.

### Description

Hi,

I'm testing Jellybench on various systems. If the system can't handle a run with a speed of >= 1 Jellybench crashes with the following error:

### Error Code or Output

```shell
> > > > Failed: ['performance']
| > > Current Test: 2160p - 720p
| > > > Current Device: cpu
| > > > ffmpeg command: ./jellybench_data/ffmpeg/ffmpeg_files/ffmpeg -c:v h264 -i /home/redacted/jellybench_py/jellybench_data/videos/Test%20Jellyfin%204K%20AVC%20150M.mp4 -noautoscale -an -sn -vf "scale=trunc(min(max(iw\,ih*a)\,528)/2)*2:trunc(ow/a/2)*2,format=yuv420p" -c:v libx265 -preset veryfast -b:v 3616000 -maxrate 3616000 -f null - -benchmark
| > > > > Starting run with 1 Workers... completed with speed 0.99
| > > > > Failed: ['limited']
Traceback (most recent call last):
File "/home/redacted/jellybench_py/bin/jellybench", line 6, in
sys.exit(main())
^^^^^^
File "/home/redacted/jellybench_py/jellybench_py/core.py", line 1045, in main
return cli()
^^^^^
File "/home/redacted/jellybench_py/jellybench_py/core.py", line 998, in cli
valid, runs, result = benchmark(
^^^^^^^^^^
File "/home/redacted/jellybench_py/jellybench_py/core.py", line 360, in benchmark
"single_worker_speed": max_pass_run_data["speed"],
~~~~~~~~~~~~~~~~~^^^^^^^^^
KeyError: 'speed'
```

### Steps to Reproduce

Run Jellybench on a system that can't handle cpu intensive transcoding tasks

### Operating System

Linux

### System Architecture

x86_64 (64-bit)

### Python Version

3.12.3

### CPU[s]

Intel(R) Core(TM) i5-7500T CPU @ 2.70GHz

### GPU[s]

Intel HD Graphics 630

### Additional Context

_No response_

### Possible Solution

Hotfix is to set "speed" and "rss_kb"to 0 if not set. Don't know how the API will handle this, though. Change line 360 and 361 in core.py from

```
"single_worker_speed": max_pass_run_data["speed"],
"single_worker_rss_kb": max_pass_run_data["rss_kb"],
```

to

```
"single_worker_speed": max_pass_run_data.get("speed", 0),
"single_worker_rss_kb": max_pass_run_data.get("rss_kb", 0),
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.