abetlen / abetlen/llama-cpp-python

BUG: import error or execute error (NULL pointer access) for the latest prebuilt version `v0.2.81`

Abierto
#1,571 8 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Python
Estrellas
10.6k
Forks
1.4k
Métricas de merge de PR
Métricas de PR pendientes

Descripción

For the latest version `v0.2.81`:

1. If I install it via prebuilt channel:
```
pip install -U llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu
```
Import error happens:
```
from llama_cpp import Llama
```
Error:
```
In [1]: from llama_cpp import Llama
---------------------------------------------------------------------------
OSError Traceback (most recent call last)
File ~/miniconda3/lib/python3.9/site-packages/llama_cpp/llama_cpp.py:75, in _load_shared_library(lib_base_name)
74 try:
---> 75 return ctypes.CDLL(str(_lib_path), **cdll_args) # type: ignore
76 except Exception as e:

File ~/miniconda3/lib/python3.9/ctypes/__init__.py:374, in CDLL.__init__(self, name, mode, handle, use_errno, use_last_error, winmode)
373 if handle is None:
--> 374 self._handle = _dlopen(self._name, mode)
375 else:

OSError: libc.musl-x86_64.so.1: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

RuntimeError Traceback (most recent call last)
Cell In[1], line 1
----> 1 from llama_cpp import Llama

File ~/miniconda3/lib/python3.9/site-packages/llama_cpp/__init__.py:1
----> 1 from .llama_cpp import *
2 from .llama import *
4 __version__ = "0.2.81"

File ~/miniconda3/lib/python3.9/site-packages/llama_cpp/llama_cpp.py:88
85 _lib_base_name = "llama"
87 # Load the library
---> 88 _lib = _load_shared_library(_lib_base_name)
91 # ctypes sane type hint helpers
92 #
93 # - Generic Pointer and Array types
(...)
96 # NOTE: Only use these for static type checking not for runtime checks
97 # no good will come of that
99 if TYPE_CHECKING:

File ~/miniconda3/lib/python3.9/site-packages/llama_cpp/llama_cpp.py:77, in _load_shared_library(lib_base_name)
75 return ctypes.CDLL(str(_lib_path), **cdll_args) # type: ignore
76 except Exception as e:
---> 77 raise RuntimeError(f"Failed to load shared library '{_lib_path}': {e}")
79 raise FileNotFoundError(
80 f"Shared library with base name '{lib_base_name}' not found"
81 )

RuntimeError: Failed to load shared library '/home/lichengjie/miniconda3/lib/python3.9/site-packages/llama_cpp/lib/libllama.so': libc.musl-x86_64.so.1: cannot open shared object file: No such file or directory
```

2. I install prebuilt version in `Docker`, the import can be passed successfully. However, the `NULL pointer access` happend when executing inference.
The error stack:
```
File "/opt/conda/lib/python3.11/site-packages/llama_cpp/llama.py", line 1132, in _create_completion
for token in self.generate(
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/llama_cpp/llama.py", line 740, in generate
self.eval(tokens)
File "/opt/conda/lib/python3.11/site-packages/llama_cpp/llama.py", line 590, in eval
logits = np.ctypeslib.as_array(self._ctx.get_logits(), shape=(rows * cols, ))
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/numpy/ctypeslib.py", line 522, in as_array
obj = ctypes.cast(obj, p_arr_type).contents
^^^^^^^^^^^^^^^^^
ValueError: [address=0.0.0.0:37293, pid=88] NULL pointer access
```

I tried prebuilt `v0.2.77`, everything is ok.
Could you please provide some solutions for these issues? Thank you.

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.