CPU buffer size issue
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 40.3k
- Forks
- 3.7k
- PR merge metrics
- No merged PRs in 30d
Description
I used microsoft/bitnet-b1.58-2B-4T-gguf model
Setup the environment:
python setup_env.py -md ./models/BitNet-b1.58-2B-4T/ -q i2_s
usage with:
python run_inference.py -m ./models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf -p "You are a helpful assistant" -cnv -n 1024
and output
llm_load_print_meta: max token length = 256 llm_load_tensors: ggml ctx size = 0.15 MiB llm_load_tensors: CPU buffer size = 1124.81 MiB ............................... llama_new_context_with_model: n_batch is less than GGML_KQ_MASK_PAD - increasing to 32 llama_new_context_with_model: n_ctx = 2048 llama_new_context_with_model: n_batch = 32 llama_new_context_with_model: n_ubatch = 32 llama_new_context_with_model: flash_attn = 0 llama_new_context_with_model: freq_base = 500000.0 llama_new_context_with_model: freq_scale = 1 llama_kv_cache_init: CPU KV buffer size = 150.00 MiB llama_new_context_with_model: KV self size = 150.00 MiB, K (f16): 75.00 MiB, V (f16): 75.00 MiB llama_new_context_with_model: CPU output buffer size = 0.49 MiB llama_new_context_with_model: CPU compute buffer size = 15.97 MiB llama_new_context_with_model: graph nodes = 1116 llama_new_context_with_model: graph splits = 1 common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable) main: llama threadpool init, n_threads = 2
My CPU buffer size is 1124.81 MiB
Why CPU usage is significantly higher than 0.4GB?
Is it an issue with my cmd args?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the report using setup_env.py and run_inference.py with the microsoft/bitnet-b1.58-2B-4T-gguf model and the shown arguments. Compare the reported CPU buffer size with the expected memory usage and determine whether the command arguments or runtime behavior account for the difference; done means a documented explanation or confirmed correction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100