GEMM tuners: an interrupted run loses everything (results are only written at the end)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 565
- Forks
- 585
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 366
Description
Tuners write -o / -o2 only after every shape completes. A long run that is interrupted — machine reclaimed, container stopped, OOM, timeout — loses all results, including the shapes that had already finished.
On a shared machine this is the dominant cost of tuning: we lost multi-hour --libtype all runs repeatedly and ended up wrapping the tuner in an external chunking driver (6 shapes per invocation) purely to bound the loss.
Proposal: append each shape's winner (and its -o2 candidates) as it completes and flush; on startup, skip shape keys already present in the output file. That is consistent with how --compare/--update_improved already reads the tuned file, gives --resume for free, and makes CI's op_tune.sh restartable.
Found while tuning GLM-5.2 on gfx950 (see #5262 for a related scalability fix from the same campaign).
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the GEMM tuner code that writes the -o and -o2 output files, then compare it with the existing --compare/--update_improved reading path. Trace how shape keys are loaded and how CI invokes the tuner through op_tune.sh; done means completed shapes are appended and flushed, reruns skip keys already present, and interrupted runs retain earlier results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- performance, tooling
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100