feat(BACKEND-ROCM-F16-WEIGHTS): retain F16 dense and embedding weights on ROCm
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 423
- Forks
- 53
- Avg merge
- 20h 26m
- Merged PRs (30d)
- 310
Description
Row: BACKEND-ROCM-F16-WEIGHTS
ROCm currently expands GGUF F16 weights because ordinary matrix multiplication and embedding kernels reject F16 storage. This prevents those weights from staying in their checkpoint format on gfx1100.
Implement F16 storage support through ordinary Matmul/MatmulBT and embedding operations, preserving the existing BF16/F32 activation and output contract. Enable loader admission for dense matrix weights and embedding tables only. Keep stacked expert weights on their existing expansion path; this issue does not enable a full F16 activation runtime.
The spec must establish mixed input arithmetic from the pinned upstream sources, cover both ID widths for embeddings, and prove the default public loading and generation path reaches the new providers. Require red-first tests, physical gfx1100 correctness and memory evidence, fresh mutation review, and operator verification.
This work is independent of PR #2782. That PR expands quantized GEMM kernels. F16 RMSNorm activation support remains a distinct concern tracked by #2542. No CI changes are included.
Spec: .agents/specs/rocm-f16-weights.md (to be committed before implementation).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading .agents/specs/rocm-f16-weights.md once it is committed, then consult the pinned upstream sources for the mixed-input arithmetic contract. Trace the default loading and generation path for dense and embedding weights, while keeping stacked experts on their existing expansion path. Done means red-first tests, gfx1100 correctness and memory evidence, mutation review, and operator verification.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100