microsoft / microsoft/BitNet

ARM i2_s (1.58-bit) CPU path produces constant output; fix in pinned llama.cpp submodule

Open
#610 2 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
40.3k
Forks
3.7k
PR merge metrics
No merged PRs in 30d

Description

The ARM CPU path for i2_s (1.58-bit ternary) produces constant, prompt-independent output (@@@@) on AArch64, while the x86 AVX2 path is correct and Metal aborts on tensor type 36. This affects builds using the pinned 3rdparty/llama.cpp submodule (isHuangXin/llama.cpp @ release-bitnet-embedding-0.6b-270m).

Root cause is in the ARM fallback of ggml_vec_dot_i2_i8_s in ggml/src/ggml-cpu/quants.c: it unpacked the 2-bit weights with a sequential layout instead of the interleaved layout the dequantizer and AVX2 path use, remapped the ternary codes instead of keeping the raw 0/1/2 the caller subtracts against, and only handled one of the two vec-dot calling conventions. Confirmed by reverting one file at a time on a clean tree (reverting quants.c brings the constant output back).

Fix (2 files, quants.c + ggml-cpu.c) submitted upstream to the pinned fork:
https://github.com/isHuangXin/llama.cpp/pull/6

Tested on Apple M2, same i2_s GGUF before and after: coherent prompt-dependent output restored (5 of 5 distinct prompts distinct), about 30 tok/s generation vs about 2 tok/s on the scalar path, and 48 of 48 exact-integer parity against an independent scalar reference. This lines up with the open issues #468 (to_float UB) and #585 (i2_s layout).

Credit: the degeneracy was first isolated in a two-node verification pass by John Reed (github.com/jar9, Meridian North). Once the submodule pin picks up the fix by commit, the BitNet ARM CPU path is restored.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with ggml/src/ggml-cpu/quants.c and ggml/src/ggml-cpu/ggml-cpu.c, then review upstream pull request #6 and reproduce the ARM i2_s case on the pinned llama.cpp submodule. Compare prompt-dependent output and exact-integer parity with the scalar reference; done means the submodule pin includes the fix and the ARM path no longer produces constant output.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.