ARM i2_s (1.58-bit) CPU path produces constant output; fix in pinned llama.cpp submodule
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 40.3k
- Forks
- 3.7k
- PR merge metrics
- No merged PRs in 30d
Description
The ARM CPU path for i2_s (1.58-bit ternary) produces constant, prompt-independent output (@@@@) on AArch64, while the x86 AVX2 path is correct and Metal aborts on tensor type 36. This affects builds using the pinned 3rdparty/llama.cpp submodule (isHuangXin/llama.cpp @ release-bitnet-embedding-0.6b-270m).
Root cause is in the ARM fallback of ggml_vec_dot_i2_i8_s in ggml/src/ggml-cpu/quants.c: it unpacked the 2-bit weights with a sequential layout instead of the interleaved layout the dequantizer and AVX2 path use, remapped the ternary codes instead of keeping the raw 0/1/2 the caller subtracts against, and only handled one of the two vec-dot calling conventions. Confirmed by reverting one file at a time on a clean tree (reverting quants.c brings the constant output back).
Fix (2 files, quants.c + ggml-cpu.c) submitted upstream to the pinned fork:
https://github.com/isHuangXin/llama.cpp/pull/6
Tested on Apple M2, same i2_s GGUF before and after: coherent prompt-dependent output restored (5 of 5 distinct prompts distinct), about 30 tok/s generation vs about 2 tok/s on the scalar path, and 48 of 48 exact-integer parity against an independent scalar reference. This lines up with the open issues #468 (to_float UB) and #585 (i2_s layout).
Credit: the degeneracy was first isolated in a two-node verification pass by John Reed (github.com/jar9, Meridian North). Once the submodule pin picks up the fix by commit, the BitNet ARM CPU path is restored.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with ggml/src/ggml-cpu/quants.c and ggml/src/ggml-cpu/ggml-cpu.c, then review upstream pull request #6 and reproduce the ARM i2_s case on the pinned llama.cpp submodule. Compare prompt-dependent output and exact-integer parity with the scalar reference; done means the submodule pin includes the fix and the ARM path no longer produces constant output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100