microsoft / microsoft/BitNet

I2_S produces incorrect output on aarch64

Open
#598 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
40.3k
Forks
3.7k
PR merge metrics
No merged PRs in 30d

Description

I2_S inference is broken on aarch64. The model loads and runs, but output is nonsense — it does not crash, so it looks like a bad model rather than a broken kernel.

On an Orange Pi 5 Plus (RK3588, Debian 12, GCC 12.2) with microsoft/BitNet-b1.58-2B-4T's official ggml-model-i2_s.gguf:

$ llama-cli -m ggml-model-i2_s.gguf -p "The capital of France is" -n 12 --temp 0
> The capital of France is  ????????????????
[ Prompt: 0.7 t/s | Generation: 0.7 t/s ]

The same file on x86-64 gives The capital of France is Paris. at ~40 t/s. Model md5 verified identical on both machines.

Three separate bugs, all in 3rdparty/llama.cpp code paths that x86 never compiles:

  1. QK_I2_S is 128 under AVX2 but 64 under __ARM_NEON, in both quants.c and ggml-cpu-i2s.c. It is the on-disk block size, so it must match the file format on every architecture.
  2. The scalar vec_dot fallback decodes the block-interleaved weight layout sequentially.
  3. ggml_gemm_i2_i8_s's ACT_PARALLEL branch inverts ggml_vec_dot_i2_i8_s's nrc semantics, corrupting prefill.

There is also no NEON path for I2_S at all — aarch64 unpacks one 2-bit weight at a time.

Fixes in https://github.com/isHuangXin/llama.cpp/pull/2, against the pinned 3rdparty/llama.cpp submodule. After them, perplexity on aarch64 matches x86 to 0.161% (74.0952 vs 73.9758, same model and corpus), and kernel output is bit-identical for both GEMV and GEMM.

This may be the same root cause as #55.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in the pinned 3rdparty/llama.cpp submodule, focusing on quants.c, ggml-cpu-i2s.c, ggml_gemm_i2_i8_s, and ggml_vec_dot_i2_i8_s. Reproduce with the listed llama-cli command on aarch64, then compare perplexity and GEMV/GEMM kernel output against x86. Done means matching output and perplexity across architectures.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.