lablup / lablup/mlxcel

chore(rocm): mlxcelverse ROCm fork sync script, MLX pin-bump procedure, and upstreaming local fixes

Open
#1,813 0 comments 0 reactions 0 assignees View on GitHub
area:core platform:linux priority:medium status:backlog type:chore
Dominant language
Rust
Stars
467
Forks
54
Avg merge
4h 25m
Merged PRs (30d)
310

Description

Part of #1801. Phase 3. Depends on #1802.

## Context

The ROCm part of mlxcelverse has two moving upstreams:

1. `ml-explore/mlx`, which mlxcel bumps regularly. Each bump can change the 15 core overlay files and the APIs the 106 backend files compile against. The retarget in #1801 needed six fixes after 360 upstream commits (a host-only `isnan` template hiding device overloads, a changed return tuple, constructor argument changes, a new `force_fused` parameter, new primitives needing `NO_GPU` stubs, and `Event::error`). Because `libmlx` is static in mlxcel, a missing `eval_gpu` shows up at link time.
2. `NripeshN/mlx@rocm-support`, where ROCm kernels improve quickly (dozens of commits in July 2026 alone: WMMA flash attention, MoE, allocator).

Whole-file overlays carry a known review risk: the existing bump check (the overlay's +/- line counts against the new upstream base must equal those against the old base) does not catch an overlay that was already wrong before the bump. A Metal overlay once carried only half of an upstream fix for months until a reviewer compared it line by line.

Local fixes made while vendoring should flow back to the fork so the overlay does not drift from its source.

## Scope

Tooling and written procedure so that both kinds of update are routine, plus sending the local fixes upstream.

## Implementation plan

1. `scripts/mlxcelverse/sync_from_fork.sh `: fetch the fork, copy `mlx/backend/rocm/` into `patches-rocm/`, re-apply the local fix list, and for each of the 15 core files produce the fork's diff against its own merge base so it can be 3-way merged (`git merge-file`) into the current overlay. Update `patches-rocm/UPSTREAM`.
2. `scripts/mlxcelverse/check_api_drift.sh`: build the ROCm tree at a candidate pin and list undefined `mlx::core` symbols and compile errors, so a bump PR sees the ROCm breakage immediately.
3. Extend the MLX pin-bump procedure in `CONTRIBUTING.md` (lines 127-153) and the technical report template referenced there with a ROCm section: 3-way merge each core overlay, compare each overlay line by line with upstream (not only line counts), re-run the ROCm op checks and the smoke from #1811, and record the drift fixes in the PR body.
4. Keep a `patches-rocm/LOCAL_FIXES.md` listing every change relative to the fork commit, with the reason and an upstream link once sent.
5. Open PRs against `NripeshN/mlx` (`rocm-support`) for the fixes that apply to the fork itself: the E8M0 `uint8_t` scale dispatch for mxfp4/mxfp8 qmv, and any kernel fixes from #1804 and #1808. Retarget-only fixes (the `isnan` qualification, API drift) go upstream when the fork itself merges newer MLX.

## Acceptance criteria

- [ ] Running the sync script against the current fork commit reproduces the committed `patches-rocm/` byte for byte.
- [ ] The drift check lists the six known breaks when run against the fork's original merge base plus current upstream.
- [ ] `CONTRIBUTING.md` documents the ROCm part of a pin bump.
- [ ] `LOCAL_FIXES.md` exists; the scale-dispatch fix is proposed upstream with a link recorded.

## References

- `CONTRIBUTING.md:127-153`, `TECHNICAL_REPORTS/1772-mlx-pin-mxfp8-round-up-20260911.en.md` (lines 61-79)
- Fork: https://github.com/NripeshN/mlx/tree/rocm-support

Contributor guide

Open the contributing guide

Research direction

Start with scripts/mlxcelverse/sync_from_fork.sh and check_api_drift.sh, then read CONTRIBUTING.md:127-153 and the referenced technical report. Review the existing patches-rocm/ state and fork commit before designing the sync and drift checks. Done means the scripts reproduce the overlay, report the known breaks, the ROCm bump procedure is documented, and LOCAL_FIXES.md records the upstream proposal.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, shell
Domain
build-system, documentation, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.