feat(BACKEND-ROCM-QUANT-GATHER): gather quantized embeddings on ROCm
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 423
- Forks
- 53
- Avg merge
- 20h 26m
- Merged PRs (30d)
- 310
Description
Row: BACKEND-ROCM-QUANT-GATHER
ROCm has no native kEmbeddingQuant provider. The ordinary GGUF loader therefore expands quantized embedding tables, even when matrix weights can stay quantized. This consumes memory and leaves a backend capability gap on gfx1100.
Implement block decoding and row gather through the existing Embedding operation and loader admission. Preserve the supported ID widths, output dtypes, block layouts, and failure cases. Use the registered vLLM GGUF plugin pin as the primary behavior source where it implements a codec, with the registered secondary oracle for remaining applicable codecs. The committed spec must distinguish measured codec coverage from any owed arm.
Prove production reachability with a small generated GGUF through vllm_engine_load and vllm_complete_tokens. Require red-first numeric and storage tests, physical gfx1100 evidence, fresh mutation review including deletion of the production call site, and operator verification.
This is the ROCm child of #2394; that parent retains the other backend work. It does not depend on the quantized GEMM provider added by PR #2782. No CI changes are included.
Spec: .agents/specs/rocm-quant-gather.md, committed before implementation at 670e6d78ddf55231394748e0032939fd53dc56a5.
The operator approved a scoped Q8_K materializer amendment on 9 September 2026 UTC. The native public test reaches Q8_K and its load returns status 2. The new loader-focused test identifies the earlier reader refusal: GGML type 15 has no FindGgmlTraits entry. A separate downstream source hypothesis is that OwnGgufQuantBlocks requires a dot kernel after the gather policy admits the decoder-only format. Add an explicit embedding role at that call, preserve the default matrix dot requirement, and test both loader reachability and matrix refusal. The row spec commits this amendment before either shared materializer file changes. No shared dtype set, quantized dot provider, or PR #2782 work changes.
The operator approved the reader amendment before implementation: add only the existing Q8_K type-15 geometry, 256 elements and 292 bytes, preserving all other traits and validation. Test exact packed bytes through the reader, rerun the loader red to identify the downstream materializer refusal, and mutate both the type entry and geometry. The scope remains this row’s declared 19-codec public path.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with .agents/specs/rocm-quant-gather.md and the existing Embedding operation and loader admission paths. Use vllm_engine_load and vllm_complete_tokens to establish production reachability, then run the red-first reader, loader, numeric, storage, and gfx1100 checks described in the issue. Done means the supported codec path, Q8_K reader geometry, failure cases, mutation review, and operator verification are covered without changing shared dtype or GEMM scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100