microsoft / microsoft/BitNet

Inference fails on Windows with non-AVX CPU (Intel N6000)

Open
#305 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
40.3k
Forks
3.7k
PR merge metrics
No merged PRs in 30d

Description

Hello,

I am attempting to run the BitNet model on a Windows 11 machine with an Intel N6000 CPU, which does not have AVX/AVX2 support. The installation completes, but inference results in a repeating character output (e.g., "GGGGGG...").

Key Findings:

  • This behavior is reproducible on my Intel N6000 machine.
  • I can successfully compile and run the same model on a Raspberry Pi 4 B, which proves that AVX is not a fundamental requirement for the model's logic. This suggests the bug is specific to the Windows x86 non-AVX build.

Steps to Reproduce:

  1. On a Windows machine with a non-AVX CPU (e.g., Intel N6000), follow the standard installation instructions.
  2. During the build process, a compilation error occurs in 3rdparty/llama.cpp/common/common.cpp due to a missing header. Adding #include <chrono> fixes this initial error.
  3. The project then compiles successfully.
  4. Running inference with a command like python run_inference.py -m models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf -p "Once upon a time" results in a repeating character output.

What I've Tried:

  • Compiling with the default settings.
  • Forcing a build with -DLLAMA_SSE4_2=ON.
  • Forcing a generic build with no flags.

All of these configurations compile successfully but produce the same incorrect inference output. The system_info log confirms that AVX is disabled.

This seems to be a bug in the x86 fallback code path when compiled with the Windows toolchain.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with 3rdparty/llama.cpp/common/common.cpp and the x86 fallback path, then reproduce through run_inference.py using the reported BitNet model and prompt on a non-AVX Windows CPU. Compare the system_info output and inference behavior with the Raspberry Pi 4 build; done means the Windows N6000 produces normal text instead of repeating characters without regressing supported builds.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.