abetlen / abetlen/llama-cpp-python

mpt-7b-storywrier-4bit-km-gguf-file -- not loading - missing one tensor

オープン
#1,251 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
Python
スター
10.6k
フォーク
1.4k
PR マージ指標
PR 指標を取得中

説明

Trying to upload 4 bit K_M gguf file from:
hf_hub_download(
repo_id="maddes8cht/mosaicml-mpt-7b-storywriter-gguf",
filename="mosaicml-mpt-7b-storywriter-Q4_K_M.gguf",
local_dir="/home/ubuntu/llms_dump",
local_dir_use_symlinks=False
)
I am able to download the file.

The model is loading up when I instantiate Llama class:

llm_mptsw7b=Llama(model_path="/home/ubuntu/llms_dump/mosaicml-mpt-7b-storywriter-Q4_K_M.gguf",n_ctx=65536,offload_kqv=True)

it is throwing in the error:
llama_model_load: error loading model: done_getting_tensors: wrong number of tensors; expected 195, got 194
llama_load_model_from_file: failed to load model
ValueError: Failed to load model from file: /home/ubuntu/llms_dump/mosaicml-mpt-7b-storywriter-Q4_K_M.gguf

llama_model_loader: - type f32: 65 tensors
llama_model_loader: - type q4_K: 81 tensors
llama_model_loader: - type q5_K: 32 tensors
llama_model_loader: - type q6_K: 17 tensors

I do not which tensor among these 195 tensors are being missed.

code to reproduce:

>>> hf_hub_download(repo_id="maddes8cht/mosaicml-mpt-7b-storywriter-gguf", filename="mosaicml-mpt-7b-storywriter-Q4_K_M.gguf",local_dir="/home/ubuntu/llms_dump",local_dir_use_symlinks=False)
mosaicml-mpt-7b-storywriter-Q4_K_M.gguf: 100%|██████████████████████████████████████████████████████| 4.39G/4.39G [00:49<00:00, 89.2MB/s]
'/home/ubuntu/llms_dump/mosaicml-mpt-7b-storywriter-Q4_K_M.gguf'
>>> os.listdir("/home/ubuntu/llms_dump")
['mosaicml-mpt-7b-storywriter-Q4_K_M.gguf']
>>> llm_mptsw7b=Llama(model_path="/home/ubuntu/llms_dump/mosaicml-mpt-7b-storywriter-Q4_K_M.gguf",n_ctx=65536,offload_kqv=True)

Output:
ggml_init_cublas: GGML_CUDA_FORCE_MMQ: no
ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes
ggml_init_cublas: found 1 CUDA devices:
Device 0: NVIDIA A10G, compute capability 8.6, VMM: yes
llama_model_loader: loaded meta data with 19 key-value pairs and 195 tensors from /home/ubuntu/llms_dump/mosaicml-mpt-7b-storywriter-Q4_K_M.gguf (version GGUF V3 (latest))

File "/opt/conda/lib/python3.10/site-packages/llama_cpp/llama.py", line 311, in __init__
self._model = _LlamaModel(
File "/opt/conda/lib/python3.10/site-packages/llama_cpp/_internals.py", line 55, in __init__
raise ValueError(f"Failed to load model from file: {path_model}")
ValueError: Failed to load model from file: /home/ubuntu/llms_dump/mosaicml-mpt-7b-storywriter-Q4_K_M.gguf

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。