microsoft / microsoft/BitNet

build/bin/llama-cli throws SIGSEGV, Segmentation fault when using a TL2 GGUF model

Open
#631 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
40.3k
Forks
3.7k
PR merge metrics
No merged PRs in 30d

Description

To reproduce on a x86 architecture:

  1. python3 setup_env.py --hf-repo HF1BitLLM/Llama3-8B-1.58-100B-tokens -q tl2
  2. python3 run_inference.py -m models/bitnet_b1_58-large/ggml-model-tl2.gguf -p "Hi are you a LLM?" -n 6 -temp 0

Result:
Loading model... |Error occurred while running command: Command '['build/bin/llama-cli', '-m', 'models/bitnet_b1_58-large/ggml-model-tl2.gguf', '-n', '6', '-t', '2', '-p', 'Hi are you a LLM?', '-ngl', '0', '-c', '2048', '--temp', '0.0']' died with <Signals.SIGSEGV: 11>.

Compiling llama.cpp in debug and run llama-cli in gdb I get this back trace:
#0 0x0000000000000000 in ?? ()
#1 0x00007ffff5e0bde9 in ggml_compute_forward_mul_mat_one_chunk (params=0x7ffffffee9a0, dst=0x23f0730, type=GGML_TYPE_TL2, num_rows_per_vec_dot=0,
ir0_start=0, ir0_end=16, ir1_start=0, ir1_end=2) at /home/hbaastrup/bitnet/BitNet.debug/3rdparty/llama.cpp/ggml/src/ggml-cpu/ggml-cpu.c:1295
#2 0x00007ffff5e0ad40 in ggml_compute_forward_mul_mat (params=0x7ffffffee9a0, dst=0x23f0730)
at /home/hbaastrup/bitnet/BitNet.debug/3rdparty/llama.cpp/ggml/src/ggml-cpu/ggml-cpu.c:1608
...

When I take a better look at the file 3rdparty/llama.cpp/ggml/src/ggml-cpu/ggml-cpu.c, I see the type_traits_cpu array does not have an value for the index of GGML_TYPE_TL2 (42), which means; when the program try to lookup the vec_dot reference, it get a 0 (null) there will provoke a SIGSEGV when vec_dot is called.

Is there a quick fix for this?

Note
I don't use the I2_S format as llama-quantize does not support this format any longer. I get:
llama_quantize: invalid ftype 'I2_S'
when I run python3 setup_env.py --hf-repo HF1BitLLM/Llama3-8B-1.58-100B-tokens -q i2_s

Looking in the file 3rdparty/llama.cpp/tools/quantize/quantize.cpp I can see the std::vector QUANT_OPTIONS missing an index for I2_S

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce with the listed setup_env.py and run_inference.py commands, then inspect ggml/src/ggml-cpu/ggml-cpu.c around ggml_compute_forward_mul_mat and the type_traits_cpu array. Also check tools/quantize/quantize.cpp and its QUANT_OPTIONS entry for I2_S. Done means the TL2 GGUF inference no longer segfaults and the reported I2_S quantization option is handled consistently.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.