Inference fails on Windows with non-AVX CPU (Intel N6000)
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 40.3k
- Forks
- 3.7k
- PR merge metrics
- No merged PRs in 30d
Description
Hello,
I am attempting to run the BitNet model on a Windows 11 machine with an Intel N6000 CPU, which does not have AVX/AVX2 support. The installation completes, but inference results in a repeating character output (e.g., "GGGGGG...").
Key Findings:
- This behavior is reproducible on my Intel N6000 machine.
- I can successfully compile and run the same model on a Raspberry Pi 4 B, which proves that AVX is not a fundamental requirement for the model's logic. This suggests the bug is specific to the Windows x86 non-AVX build.
Steps to Reproduce:
- On a Windows machine with a non-AVX CPU (e.g., Intel N6000), follow the standard installation instructions.
- During the build process, a compilation error occurs in
3rdparty/llama.cpp/common/common.cppdue to a missing header. Adding#include <chrono>fixes this initial error. - The project then compiles successfully.
- Running inference with a command like
python run_inference.py -m models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf -p "Once upon a time"results in a repeating character output.
What I've Tried:
- Compiling with the default settings.
- Forcing a build with
-DLLAMA_SSE4_2=ON. - Forcing a generic build with no flags.
All of these configurations compile successfully but produce the same incorrect inference output. The system_info log confirms that AVX is disabled.
This seems to be a bug in the x86 fallback code path when compiled with the Windows toolchain.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with 3rdparty/llama.cpp/common/common.cpp and the x86 fallback path, then reproduce through run_inference.py using the reported BitNet model and prompt on a non-AVX Windows CPU. Compare the system_info output and inference behavior with the Raspberry Pi 4 build; done means the Windows N6000 produces normal text instead of repeating characters without regressing supported builds.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100