abetlen / abetlen/llama-cpp-python

mpt-7b-storywrier-4bit-km-gguf-file -- not loading - missing one tensor

Đang mở
#1,251 1 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Python
Star
10.6k
Fork
1.4k
Chỉ số merge pull request
Chỉ số pull request đang chờ

Mô tả

Trying to upload 4 bit K_M gguf file from:
hf_hub_download(
repo_id="maddes8cht/mosaicml-mpt-7b-storywriter-gguf",
filename="mosaicml-mpt-7b-storywriter-Q4_K_M.gguf",
local_dir="/home/ubuntu/llms_dump",
local_dir_use_symlinks=False
)
I am able to download the file.

The model is loading up when I instantiate Llama class:

llm_mptsw7b=Llama(model_path="/home/ubuntu/llms_dump/mosaicml-mpt-7b-storywriter-Q4_K_M.gguf",n_ctx=65536,offload_kqv=True)

it is throwing in the error:
llama_model_load: error loading model: done_getting_tensors: wrong number of tensors; expected 195, got 194
llama_load_model_from_file: failed to load model
ValueError: Failed to load model from file: /home/ubuntu/llms_dump/mosaicml-mpt-7b-storywriter-Q4_K_M.gguf

llama_model_loader: - type f32: 65 tensors
llama_model_loader: - type q4_K: 81 tensors
llama_model_loader: - type q5_K: 32 tensors
llama_model_loader: - type q6_K: 17 tensors

I do not which tensor among these 195 tensors are being missed.

code to reproduce:

>>> hf_hub_download(repo_id="maddes8cht/mosaicml-mpt-7b-storywriter-gguf", filename="mosaicml-mpt-7b-storywriter-Q4_K_M.gguf",local_dir="/home/ubuntu/llms_dump",local_dir_use_symlinks=False)
mosaicml-mpt-7b-storywriter-Q4_K_M.gguf: 100%|██████████████████████████████████████████████████████| 4.39G/4.39G [00:49<00:00, 89.2MB/s]
'/home/ubuntu/llms_dump/mosaicml-mpt-7b-storywriter-Q4_K_M.gguf'
>>> os.listdir("/home/ubuntu/llms_dump")
['mosaicml-mpt-7b-storywriter-Q4_K_M.gguf']
>>> llm_mptsw7b=Llama(model_path="/home/ubuntu/llms_dump/mosaicml-mpt-7b-storywriter-Q4_K_M.gguf",n_ctx=65536,offload_kqv=True)

Output:
ggml_init_cublas: GGML_CUDA_FORCE_MMQ: no
ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes
ggml_init_cublas: found 1 CUDA devices:
Device 0: NVIDIA A10G, compute capability 8.6, VMM: yes
llama_model_loader: loaded meta data with 19 key-value pairs and 195 tensors from /home/ubuntu/llms_dump/mosaicml-mpt-7b-storywriter-Q4_K_M.gguf (version GGUF V3 (latest))

File "/opt/conda/lib/python3.10/site-packages/llama_cpp/llama.py", line 311, in __init__
self._model = _LlamaModel(
File "/opt/conda/lib/python3.10/site-packages/llama_cpp/_internals.py", line 55, in __init__
raise ValueError(f"Failed to load model from file: {path_model}")
ValueError: Failed to load model from file: /home/ubuntu/llms_dump/mosaicml-mpt-7b-storywriter-Q4_K_M.gguf

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.