lablup / lablup/mlxcel

perf(bench): ROCm support in the benchmark harness and a published gfx1151 baseline

Open
#1,810 0 comments 0 reactions 0 assignees View on GitHub
area:benchmark platform:linux priority:medium status:backlog type:performance
Dominant language
Rust
Stars
467
Forks
54
Avg merge
4h 25m
Merged PRs (30d)
310

Description

Part of #1801. Phase 3. Depends on #1802, #1805.

## Context

`scripts/bench_decode.sh` runs on Linux, but on an AMD host it mislabels and misjudges:

- `detect_backend` (line 297) returns `cuda` only when `nvidia-smi` works or `generate --help` mentions cuda, otherwise `metal`, so ROCm results would be written as `benchmarks/metal__.csv`.
- The hardware tag falls back to the first 20 characters of the `/proc/cpuinfo` model name.
- The out-of-memory guard uses 85% of host RAM, which is wrong on a UMA carve-out (host sees about 31 GiB, the GPU 96 GiB); the only knob is `BENCH_MEM_OVERHEAD_FACTOR`.
- `scripts/compare_bench_csv.py:107-108` hardcodes hosts (`m5max`, `m1ultra`) and runtimes (`pylm`, `metal`), so its superseded-baseline warning silently turns off for other hosts.
- The CSV `mlx_commit` column stores 8 characters and cannot tell upstream MLX from a build that carries the mlxcelverse ROCm overlay.

Spike numbers for orientation (mlx-lm on a build of the same mlxcelverse ROCm tree, pp512/tg128): Qwen3-0.6B-4bit 3,899/224, Llama-3.1-8B-Instruct-4bit 925/32.3, Qwen3-30B-A3B-4bit 295/58.7 tok/s. Decode-shaped q4 GEMV reached about 150 GB/s.

## Scope

Make the harness produce correctly labeled, comparable ROCm results, and publish a first `gfx1151` baseline.

## Implementation plan

1. `detect_backend`: return `rocm` when the binary was built with the `rocm` feature (query `generate --help` or a version/diagnostics flag from #1805) or `rocminfo` finds a GPU agent.
2. Hardware tag: recognize AMD GPUs (for example `strixhalo-gfx1151`) from the device info exposed in #1805.
3. Memory guard: use device memory from #1805 when the backend is ROCm.
4. Add a column or metadata line for the ROCm overlay's source fork commit and the ROCm/HIP version, and keep `mlx_commit` meaning the upstream pin.
5. Generalize host and runtime detection in `compare_bench_csv.py`.
6. Run `./scripts/bench_decode.sh all --cooldown 30 --big-cooldown 30` on the spike host for the models in #1809, plus the mlx-lm Python baseline built from the same mlxcelverse ROCm tree (`scripts/bench_mlxlm.py`), same day and same mlxcel commit. Publish the CSV in `benchmarks/` and a page in `docs/benchmark_results/` with hardware, OS, ROCm/HIP version, `gfx` target, mlxcel commit, MLX pin, mlxcelverse commit, checkpoint and quantization format, prompt/decode lengths, batch, warmup and flags.
7. Note in the page whether other GPU tenants were stopped during the run (the spike host also runs GPU services).

## Acceptance criteria

- [ ] ROCm runs are written as `benchmarks/rocm__.csv` with correct backend and hardware tags.
- [ ] The memory guard does not skip models that fit in device memory on the UMA host.
- [ ] `compare_bench_csv.py` handles the new host and runtime without disabling its warnings.
- [ ] A `gfx1151` baseline page with mlxcel versus mlx-lm on the same build is published.

## References

- `scripts/bench_decode.sh` (backend detection 297-309, GPU name 249), `scripts/compare_bench_csv.py:107-108`, `scripts/bench_mlxlm.py`
- Existing result pages: `docs/benchmark_results/`

Contributor guide

Open the contributing guide

Research direction

Start by reading scripts/bench_decode.sh around backend detection and GPU naming, then scripts/compare_bench_csv.py:107-108 and the device information from #1805; review scripts/bench_mlxlm.py and existing pages in docs/benchmark_results/. Run the specified benchmark command on the spike host, and consider the work done when ROCm CSV labels, memory handling, comparison warnings, and the gfx1151 mlxcel-versus-mlx-lm baseline page meet the acceptance criteria.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust, shell
Domain
documentation, performance, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.