ROCm / ROCm/aiter

GEMM tuners: an interrupted run loses everything (results are only written at the end)

Open
#5,263 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
565
Forks
585
Avg merge
3d 4h
Merged PRs (30d)
366

Description

Tuners write -o / -o2 only after every shape completes. A long run that is interrupted — machine reclaimed, container stopped, OOM, timeout — loses all results, including the shapes that had already finished.

On a shared machine this is the dominant cost of tuning: we lost multi-hour --libtype all runs repeatedly and ended up wrapping the tuner in an external chunking driver (6 shapes per invocation) purely to bound the loss.

Proposal: append each shape's winner (and its -o2 candidates) as it completes and flush; on startup, skip shape keys already present in the output file. That is consistent with how --compare/--update_improved already reads the tuned file, gives --resume for free, and makes CI's op_tune.sh restartable.

Found while tuning GLM-5.2 on gfx950 (see #5262 for a related scalability fix from the same campaign).

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the GEMM tuner code that writes the -o and -o2 output files, then compare it with the existing --compare/--update_improved reading path. Trace how shape keys are loaded and how CI invokes the tuner through op_tune.sh; done means completed shapes are appended and flushed, reruns skip keys already present, and interrupted runs retain earlier results.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.