perf(bench): ROCm support in the benchmark harness and a published gfx1151 baseline
- Dominant language
- Rust
- Stars
- 467
- Forks
- 54
- Avg merge
- 4h 25m
- Merged PRs (30d)
- 310
Description
Part of #1801. Phase 3. Depends on #1802, #1805.
## Context
`scripts/bench_decode.sh` runs on Linux, but on an AMD host it mislabels and misjudges:
- `detect_backend` (line 297) returns `cuda` only when `nvidia-smi` works or `generate --help` mentions cuda, otherwise `metal`, so ROCm results would be written as `benchmarks/metal__.csv`.
- The hardware tag falls back to the first 20 characters of the `/proc/cpuinfo` model name.
- The out-of-memory guard uses 85% of host RAM, which is wrong on a UMA carve-out (host sees about 31 GiB, the GPU 96 GiB); the only knob is `BENCH_MEM_OVERHEAD_FACTOR`.
- `scripts/compare_bench_csv.py:107-108` hardcodes hosts (`m5max`, `m1ultra`) and runtimes (`pylm`, `metal`), so its superseded-baseline warning silently turns off for other hosts.
- The CSV `mlx_commit` column stores 8 characters and cannot tell upstream MLX from a build that carries the mlxcelverse ROCm overlay.
Spike numbers for orientation (mlx-lm on a build of the same mlxcelverse ROCm tree, pp512/tg128): Qwen3-0.6B-4bit 3,899/224, Llama-3.1-8B-Instruct-4bit 925/32.3, Qwen3-30B-A3B-4bit 295/58.7 tok/s. Decode-shaped q4 GEMV reached about 150 GB/s.
## Scope
Make the harness produce correctly labeled, comparable ROCm results, and publish a first `gfx1151` baseline.
## Implementation plan
1. `detect_backend`: return `rocm` when the binary was built with the `rocm` feature (query `generate --help` or a version/diagnostics flag from #1805) or `rocminfo` finds a GPU agent.
2. Hardware tag: recognize AMD GPUs (for example `strixhalo-gfx1151`) from the device info exposed in #1805.
3. Memory guard: use device memory from #1805 when the backend is ROCm.
4. Add a column or metadata line for the ROCm overlay's source fork commit and the ROCm/HIP version, and keep `mlx_commit` meaning the upstream pin.
5. Generalize host and runtime detection in `compare_bench_csv.py`.
6. Run `./scripts/bench_decode.sh all --cooldown 30 --big-cooldown 30` on the spike host for the models in #1809, plus the mlx-lm Python baseline built from the same mlxcelverse ROCm tree (`scripts/bench_mlxlm.py`), same day and same mlxcel commit. Publish the CSV in `benchmarks/` and a page in `docs/benchmark_results/` with hardware, OS, ROCm/HIP version, `gfx` target, mlxcel commit, MLX pin, mlxcelverse commit, checkpoint and quantization format, prompt/decode lengths, batch, warmup and flags.
7. Note in the page whether other GPU tenants were stopped during the run (the spike host also runs GPU services).
## Acceptance criteria
- [ ] ROCm runs are written as `benchmarks/rocm__.csv` with correct backend and hardware tags.
- [ ] The memory guard does not skip models that fit in device memory on the UMA host.
- [ ] `compare_bench_csv.py` handles the new host and runtime without disabling its warnings.
- [ ] A `gfx1151` baseline page with mlxcel versus mlx-lm on the same build is published.
## References
- `scripts/bench_decode.sh` (backend detection 297-309, GPU name 249), `scripts/compare_bench_csv.py:107-108`, `scripts/bench_mlxlm.py`
- Existing result pages: `docs/benchmark_results/`
Contributor guide
Research direction
Start by reading scripts/bench_decode.sh around backend detection and GPU naming, then scripts/compare_bench_csv.py:107-108 and the device information from #1805; review scripts/bench_mlxlm.py and existing pages in docs/benchmark_results/. Run the specified benchmark command on the spike host, and consider the work done when ROCm CSV labels, memory handling, comparison warnings, and the gfx1151 mlxcel-versus-mlx-lm baseline page meet the acceptance criteria.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust, shell
- Domain
- documentation, performance, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100