abetlen / abetlen/llama-cpp-python
mpt-7b-storywrier-4bit-km-gguf-file -- not loading - missing one tensor
- Dominant language
- Python
- Stars
- 10.6k
- Forks
- 1.4k
- PR merge metrics
- PR metrics pending
Description
Trying to upload 4 bit K_M gguf file from:
hf_hub_download(
repo_id="maddes8cht/mosaicml-mpt-7b-storywriter-gguf",
filename="mosaicml-mpt-7b-storywriter-Q4_K_M.gguf",
local_dir="/home/ubuntu/llms_dump",
local_dir_use_symlinks=False
)
I am able to download the file.
The model is loading up when I instantiate Llama class:
llm_mptsw7b=Llama(model_path="/home/ubuntu/llms_dump/mosaicml-mpt-7b-storywriter-Q4_K_M.gguf",n_ctx=65536,offload_kqv=True)
it is throwing in the error:
llama_model_load: error loading model: done_getting_tensors: wrong number of tensors; expected 195, got 194
llama_load_model_from_file: failed to load model
ValueError: Failed to load model from file: /home/ubuntu/llms_dump/mosaicml-mpt-7b-storywriter-Q4_K_M.gguf
llama_model_loader: - type f32: 65 tensors
llama_model_loader: - type q4_K: 81 tensors
llama_model_loader: - type q5_K: 32 tensors
llama_model_loader: - type q6_K: 17 tensors
I do not which tensor among these 195 tensors are being missed.
code to reproduce:
>>> hf_hub_download(repo_id="maddes8cht/mosaicml-mpt-7b-storywriter-gguf", filename="mosaicml-mpt-7b-storywriter-Q4_K_M.gguf",local_dir="/home/ubuntu/llms_dump",local_dir_use_symlinks=False)
mosaicml-mpt-7b-storywriter-Q4_K_M.gguf: 100%|██████████████████████████████████████████████████████| 4.39G/4.39G [00:49<00:00, 89.2MB/s]
'/home/ubuntu/llms_dump/mosaicml-mpt-7b-storywriter-Q4_K_M.gguf'
>>> os.listdir("/home/ubuntu/llms_dump")
['mosaicml-mpt-7b-storywriter-Q4_K_M.gguf']
>>> llm_mptsw7b=Llama(model_path="/home/ubuntu/llms_dump/mosaicml-mpt-7b-storywriter-Q4_K_M.gguf",n_ctx=65536,offload_kqv=True)
Output:
ggml_init_cublas: GGML_CUDA_FORCE_MMQ: no
ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes
ggml_init_cublas: found 1 CUDA devices:
Device 0: NVIDIA A10G, compute capability 8.6, VMM: yes
llama_model_loader: loaded meta data with 19 key-value pairs and 195 tensors from /home/ubuntu/llms_dump/mosaicml-mpt-7b-storywriter-Q4_K_M.gguf (version GGUF V3 (latest))
File "/opt/conda/lib/python3.10/site-packages/llama_cpp/llama.py", line 311, in __init__
self._model = _LlamaModel(
File "/opt/conda/lib/python3.10/site-packages/llama_cpp/_internals.py", line 55, in __init__
raise ValueError(f"Failed to load model from file: {path_model}")
ValueError: Failed to load model from file: /home/ubuntu/llms_dump/mosaicml-mpt-7b-storywriter-Q4_K_M.gguf
Contributor guide
Assessment
This issue has not been assessed yet.