lablup / lablup/mlxcel

fix(rocm): the WebUI catalog reports every architecture unsupported on ROCm

Open
#1,886 0 comments 0 reactions 0 assignees View on GitHub
status:backlog type:bug
Dominant language
Rust
Stars
467
Forks
54
Avg merge
4h 25m
Merged PRs (30d)
310

Description

Phase 1 ROCm work under epic #1801. Found while validating PR #1883 (issue #1805) on the ROCm spike host: AMD Ryzen AI MAX+ 395 / Radeon 8060S, `gfx1151`, RDNA 3.5, wave32, ROCm 10.0.0, Debian 13, branch `feature/issue-1805-gpu-vendor`. Pre-existing defect, not a regression from that PR.

## Context

Five tests fail in the `-p mlxcel --lib` target on a ROCm build:

- `server::webui::catalog::tests::catalog_cache_tests::catalog_cache_uses_epoch_and_projects_fresh_provider_lifecycle`, `src/server/webui/catalog_cache_tests.rs:56`
- `server::webui::catalog::tests::catalog_contract_tests::actual_catalog_producer_matches_whole_expected_shape`, `src/server/webui/catalog_contract_tests.rs:263`
- `server::webui::catalog::tests::catalog_contract_tests::cached_single_model_catalog_accessor_projects_fresh_app_state_provider`, `src/server/webui/catalog_contract_tests.rs:318`
- `server::webui::catalog::tests::catalog_contract_tests::pooling_parent_symlink_is_not_embedding_layout_evidence`, `src/server/webui/catalog_contract_tests.rs:496`
- `server::webui::catalog::tests::metadata_scan_reports_supported_complete_model_without_loading_provider`, `src/server/webui/catalog_tests.rs:117`

The contract test's diff shows a qwen3 fixture coming back with `"runnable_on_backend": false`, `"supported": false`, and every capability `"available": false` with reason `"architecture is not runnable on the current backend"` and `"runnable_on_backend_reason": "current backend does not advertise this architecture as runnable"`, where the expected shape has all of those true or null.

The defect is `runnable_on_backend` at `src/server/webui/catalog_metadata.rs:298-307`:

```rust
fn runnable_on_backend(family: &ArchitectureFamily) -> bool {
let status = if cfg!(all(target_os = "macos", feature = "metal")) {
family.backends.metal
} else if cfg!(feature = "cuda") {
family.backends.cuda
} else {
BackendStatus::Unsupported
};
matches!(status, BackendStatus::Supported | BackendStatus::Partial)
}
```

This is the same "there are two GPU backends and everything else is unsupported" assumption that #1803 removed in the kernel router and #1805 removed in hardware detection, surfacing one layer higher in the catalog. On a ROCm build both `cfg!` arms are false, so every architecture resolves to `BackendStatus::Unsupported` and the WebUI catalog tells a user that no model on the host can run, on a host where models do run. The five failing tests are the symptom; the user-visible defect is the catalog, which is why this is filed as a bug rather than a test fix.

The data the function reads has no ROCm column at all. `BackendSupport` at `src/models/registry.rs:111-114` is `{ metal: BackendStatus, cuda: BackendStatus }` and is the only consumer of the two-element arrays: `src/models/registry.rs:605-608` fills it from `caps.backends[0..1]` and `src/models/registry.rs:629-632` from `family.backends[0..1]`, both `[BackendStatus; 2]` (`src/models/registry.rs:80` and `src/models/registry.rs:159`). The source values are two sites only: `ModelType::capabilities()` at `src/models/registry.rs:469-476`, where `backends: [BackendStatus::Supported, cuda]` and `cuda` is `Partial` for `Kokoro`, `Whisper`, and `Gemma4` and `Supported` otherwise, and the single `STANDALONE_ARCHITECTURE_FAMILIES` entry (`rt_detr_v2`) at `src/models/registry.rs:176`. So the registry survey is small: one match arm and one literal, not a per-family audit. `src/server/webui/catalog_metadata.rs:300` and `:302` are the only reads of `family.backends` outside the registry, so the blast radius of widening the type is confined to these files.

Note that `BackendSupport` derives `Serialize` (`src/models/registry.rs:110`), so adding a field changes any JSON that carries it; confirm whether any endpoint or CLI output emits it before choosing the field name.

## Scope

**In scope:** `src/server/webui/catalog_metadata.rs` (the `runnable_on_backend` function), `src/models/registry.rs` (`BackendSupport`, the two `[BackendStatus; 2]` declarations, `ModelType::capabilities()`, and the `rt_detr_v2` standalone entry), and the five failing tests plus the catalog fixture they compare against.

**Out of scope:** the accuracy of per-architecture ROCm support claims beyond a first-pass value; treat the initial ROCm column as a starting point to be refined as families are actually validated on AMD, and say so in the code comment. Any other `cfg!(feature = "cuda")` two-backend assumption outside the catalog; file those separately as they are found.

## Proposed solution

Add a ROCm column to the registry data and a ROCm arm to `runnable_on_backend`, and select the arm from the runtime-resolved backend rather than from a `cfg!` chain where practical. PR #1883 adds `mlxcel_core::hardware::gpu_backend_kind()` (`src/lib/mlxcel-core/src/hardware.rs:164` on `feature/issue-1805-gpu-vendor`), which reports the backend at runtime, so the selection no longer has to be a compile-time guess. A `cfg!` chain also cannot express a build compiled with more than one GPU feature, which is the shape this bug keeps reappearing in. Keep a compile-time fallback only for the case where no GPU backend is compiled in at all.

**Coordinate before landing.** WebUI is under active concurrent development, so the catalog fixture and the contract test's expected shape are moving targets. Rebase on the current WebUI work and confirm the fixture owner's intent before changing `runnable_on_backend`'s output shape, rather than landing blind against a fixture that is about to be regenerated.

## Implementation plan

1. Add a `rocm: BackendStatus` field to `BackendSupport` at `src/models/registry.rs:111-114`, and widen `[BackendStatus; 2]` to `[BackendStatus; 3]` at `src/models/registry.rs:80` and `src/models/registry.rs:159` (or replace the positional array with a named struct, which removes the index-position coupling that made this easy to miss).
2. Populate the new column at the two source sites: the `backends:` literal in `ModelType::capabilities()` (`src/models/registry.rs:476`) and the `rt_detr_v2` entry (`src/models/registry.rs:176`). Decide the first-pass ROCm value per family and record the reasoning in a comment next to it.
3. Wire the field through the two projections at `src/models/registry.rs:605-608` and `src/models/registry.rs:629-632`.
4. Rewrite `runnable_on_backend` (`src/server/webui/catalog_metadata.rs:298`) to select the column from `mlxcel_core::hardware::gpu_backend_kind()`, mapping `Metal`, `Cuda`, and `Rocm` to their columns and falling back to `BackendStatus::Unsupported` only when no GPU backend is resolved. Keep the `matches!(status, Supported | Partial)` predicate unchanged.
5. Confirm whether `BackendSupport`'s serialized form reaches any HTTP response or CLI output; if it does, note the added field in the corresponding docs and treat it as an additive API change.
6. Regenerate or update the catalog fixture and the contract test's expected shape so the five tests assert the corrected behavior, after coordinating with in-flight WebUI work.
7. Add a regression test that pins `runnable_on_backend` per backend kind, so a future backend addition fails a test rather than silently reporting every architecture unsupported.

## Acceptance criteria

- [ ] On a ROCm build on the gfx1151 host, the WebUI catalog reports `runnable_on_backend: true` and `supported: true` for architectures the host can actually run, with `runnable_on_backend_reason` null.
- [ ] All five listed tests pass under `cargo test --workspace --profile test-fast --features rocm -p mlxcel --lib`.
- [ ] `runnable_on_backend` selects its column from the resolved backend, and a build with no GPU backend still resolves to `Unsupported`.
- [ ] `BackendSupport` carries a ROCm column populated at every construction site, with no `[BackendStatus; 2]` left.
- [ ] A test pins the per-backend-kind mapping so adding a fourth backend without a column fails.
- [ ] Metal and CUDA catalog output is byte-identical to before, verified by the unchanged contract fixture on those backends.
- [ ] The change is integrated into the catalog the WebUI actually serves, not confined to the registry type.

## Validation

```bash
cargo test --workspace --profile test-fast --features rocm -p mlxcel --lib server::webui::catalog
cargo test --workspace --profile test-fast --features rocm -p mlxcel --lib
cargo clippy --workspace --all-targets --features rocm -- -D warnings
cargo fmt --all -- --check
```

Regression guard on a CUDA host, which must stay unchanged per #1805:

```bash
cargo test --workspace --profile test-fast --features cuda -p mlxcel --lib server::webui::catalog
```

Manual check on the ROCm host: start the server, open the WebUI model catalog, and confirm a qwen3 model is listed as runnable with its capabilities available rather than every entry greyed out with "architecture is not runnable on the current backend".

## References

- Epic #1801 (AMD GPU (ROCm) backend on Linux via mlxcelverse), phase 1
- #1805 and PR #1883, where this was found and where `gpu_backend_kind()` is added
- #1803, which removed the same two-backend assumption from the kernel router

Contributor guide

Open the contributing guide

Research direction

Start with src/server/webui/catalog_metadata.rs:runnable_on_backend and src/models/registry.rs, then inspect gpu_backend_kind() at src/lib/mlxcel-core/src/hardware.rs:164 and the five listed catalog tests. Trace every BackendSupport construction and serialized use before choosing the registry shape. Done means ROCm selects the correct status, no-backend fallback remains Unsupported, all five tests and the listed validation commands pass, and CUDA/Metal output is unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.