microsoft / microsoft/BitNet

llama-server ignores bitnet-b1.58.rope.freq_base metadata, causing NaN in softmax. llama-cli works correctly.

Open
#517 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
40.3k
Forks
3.7k
PR merge metrics
No merged PRs in 30d

Description

Evidence:

  • GGUF metadata: bitnet-b1.58.rope.freq_base = 500000
  • Server crashes at ggml.c:14652 (assert !isnan) regardless of --rope-freq-base flag
  • Same model/prompt works fine with llama-cli
  • Crash appears twice simultaneously (multi-thread confirm)
  • GDB backtrace: server_context::update_slots() → llama_decode()
    → ggml_compute_forward_soft_max_f32() → __assert_fail
  • llama-cli works because it doesn't use server_context slot management

Command used:
build/bin/llama-server -m models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf -c 4096 -t 6 -n 1024 -ngl 0 -b 1 --temp 0.8 --rope-freq-base 500000 --host 127.0.0.1 --port 8080
Error:

Segmentation Fault
llama-server: /home/user/Documentos/Codigos/bitnet/bitnet_code/3rdparty/llama.cpp/ggml/src/ggml.c:14652: void ggml_compute_forward_soft_max_f32(const struct ggml_compute_params *, struct ggml_tensor *): Assertion `!isnan(wp[i])' failed.
llama-server: /home/user/Documentos/Codigos/bitnet/bitnet_code/3rdparty/llama.cpp/ggml/src/ggml.c:14652: void ggml_compute_forward_soft_max_f32(const struct ggml_compute_params *, struct ggml_tensor *): Assertion `!isnan(wp[i])' failed.

Prompt used:

---------------------
Context information is below.
[1] the positions of Russian troops  and military equipment. Information from partisans helped the Ukrainian army destroy  a  Russian warehouse in Nova Kakhovka on July 11. In September -October 2002, Ukrainian  forces intensively destroyed Russian facilities and personnel in the Kherson region,  precipita ting the rapid liberation of Kherson city and the entire Mykola yiv region in  November 2022.     Conclusion     Russian authorities have not been able to suppress Ukraine’s partisan movements or
---------------------
Query: Tell me everything you know about partisan warfare in occupied ukraine

Machine:

OS: Debian GNU/Linux 12 (bookworm) x86_64
Kernel: 6.1.0-44-amd64
CPU: Intel i7-8665U (8) @ 4.800GHz 
GPU: NVIDIA GeForce MX250 
GPU: Intel WhiskeyLake-U GT2 [UHD Graphics 620] 
Memory: 4125MiB / 7564MiB 

stacktrace.txt

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the supplied llama-server command and compare it with llama-cli using the same model and rope frequency setting. Trace server_context::update_slots() into llama_decode() and ggml_compute_forward_soft_max_f32(), using the stacktrace and the reported bitnet-b1.58.rope.freq_base metadata as context. Done means llama-server handles the model without the NaN assertion and preserves the working llama-cli behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
ai, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.