abetlen / abetlen/llama-cpp-python

mpt-7b-storywrier-4bit-km-gguf-file -- not loading - missing one tensor

Abierto
#1,251 1 comentario 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Python
Estrellas
10.6k
Forks
1.4k
Métricas de merge de PR
Métricas de PR pendientes

Descripción

Trying to upload 4 bit K_M gguf file from:
hf_hub_download(
repo_id="maddes8cht/mosaicml-mpt-7b-storywriter-gguf",
filename="mosaicml-mpt-7b-storywriter-Q4_K_M.gguf",
local_dir="/home/ubuntu/llms_dump",
local_dir_use_symlinks=False
)
I am able to download the file.

The model is loading up when I instantiate Llama class:

llm_mptsw7b=Llama(model_path="/home/ubuntu/llms_dump/mosaicml-mpt-7b-storywriter-Q4_K_M.gguf",n_ctx=65536,offload_kqv=True)

it is throwing in the error:
llama_model_load: error loading model: done_getting_tensors: wrong number of tensors; expected 195, got 194
llama_load_model_from_file: failed to load model
ValueError: Failed to load model from file: /home/ubuntu/llms_dump/mosaicml-mpt-7b-storywriter-Q4_K_M.gguf

llama_model_loader: - type f32: 65 tensors
llama_model_loader: - type q4_K: 81 tensors
llama_model_loader: - type q5_K: 32 tensors
llama_model_loader: - type q6_K: 17 tensors

I do not which tensor among these 195 tensors are being missed.

code to reproduce:

>>> hf_hub_download(repo_id="maddes8cht/mosaicml-mpt-7b-storywriter-gguf", filename="mosaicml-mpt-7b-storywriter-Q4_K_M.gguf",local_dir="/home/ubuntu/llms_dump",local_dir_use_symlinks=False)
mosaicml-mpt-7b-storywriter-Q4_K_M.gguf: 100%|██████████████████████████████████████████████████████| 4.39G/4.39G [00:49<00:00, 89.2MB/s]
'/home/ubuntu/llms_dump/mosaicml-mpt-7b-storywriter-Q4_K_M.gguf'
>>> os.listdir("/home/ubuntu/llms_dump")
['mosaicml-mpt-7b-storywriter-Q4_K_M.gguf']
>>> llm_mptsw7b=Llama(model_path="/home/ubuntu/llms_dump/mosaicml-mpt-7b-storywriter-Q4_K_M.gguf",n_ctx=65536,offload_kqv=True)

Output:
ggml_init_cublas: GGML_CUDA_FORCE_MMQ: no
ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes
ggml_init_cublas: found 1 CUDA devices:
Device 0: NVIDIA A10G, compute capability 8.6, VMM: yes
llama_model_loader: loaded meta data with 19 key-value pairs and 195 tensors from /home/ubuntu/llms_dump/mosaicml-mpt-7b-storywriter-Q4_K_M.gguf (version GGUF V3 (latest))

File "/opt/conda/lib/python3.10/site-packages/llama_cpp/llama.py", line 311, in __init__
self._model = _LlamaModel(
File "/opt/conda/lib/python3.10/site-packages/llama_cpp/_internals.py", line 55, in __init__
raise ValueError(f"Failed to load model from file: {path_model}")
ValueError: Failed to load model from file: /home/ubuntu/llms_dump/mosaicml-mpt-7b-storywriter-Q4_K_M.gguf

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.