abetlen / abetlen/llama-cpp-python

BUG: import error or execute error (NULL pointer access) for the latest prebuilt version `v0.2.81`

Open
#1,571 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
10.6k
Forks
1.4k
PR merge metrics
PR metrics pending

Description

For the latest version `v0.2.81`:

1. If I install it via prebuilt channel:
```
pip install -U llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu
```
Import error happens:
```
from llama_cpp import Llama
```
Error:
```
In [1]: from llama_cpp import Llama
---------------------------------------------------------------------------
OSError Traceback (most recent call last)
File ~/miniconda3/lib/python3.9/site-packages/llama_cpp/llama_cpp.py:75, in _load_shared_library(lib_base_name)
74 try:
---> 75 return ctypes.CDLL(str(_lib_path), **cdll_args) # type: ignore
76 except Exception as e:

File ~/miniconda3/lib/python3.9/ctypes/__init__.py:374, in CDLL.__init__(self, name, mode, handle, use_errno, use_last_error, winmode)
373 if handle is None:
--> 374 self._handle = _dlopen(self._name, mode)
375 else:

OSError: libc.musl-x86_64.so.1: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

RuntimeError Traceback (most recent call last)
Cell In[1], line 1
----> 1 from llama_cpp import Llama

File ~/miniconda3/lib/python3.9/site-packages/llama_cpp/__init__.py:1
----> 1 from .llama_cpp import *
2 from .llama import *
4 __version__ = "0.2.81"

File ~/miniconda3/lib/python3.9/site-packages/llama_cpp/llama_cpp.py:88
85 _lib_base_name = "llama"
87 # Load the library
---> 88 _lib = _load_shared_library(_lib_base_name)
91 # ctypes sane type hint helpers
92 #
93 # - Generic Pointer and Array types
(...)
96 # NOTE: Only use these for static type checking not for runtime checks
97 # no good will come of that
99 if TYPE_CHECKING:

File ~/miniconda3/lib/python3.9/site-packages/llama_cpp/llama_cpp.py:77, in _load_shared_library(lib_base_name)
75 return ctypes.CDLL(str(_lib_path), **cdll_args) # type: ignore
76 except Exception as e:
---> 77 raise RuntimeError(f"Failed to load shared library '{_lib_path}': {e}")
79 raise FileNotFoundError(
80 f"Shared library with base name '{lib_base_name}' not found"
81 )

RuntimeError: Failed to load shared library '/home/lichengjie/miniconda3/lib/python3.9/site-packages/llama_cpp/lib/libllama.so': libc.musl-x86_64.so.1: cannot open shared object file: No such file or directory
```

2. I install prebuilt version in `Docker`, the import can be passed successfully. However, the `NULL pointer access` happend when executing inference.
The error stack:
```
File "/opt/conda/lib/python3.11/site-packages/llama_cpp/llama.py", line 1132, in _create_completion
for token in self.generate(
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/llama_cpp/llama.py", line 740, in generate
self.eval(tokens)
File "/opt/conda/lib/python3.11/site-packages/llama_cpp/llama.py", line 590, in eval
logits = np.ctypeslib.as_array(self._ctx.get_logits(), shape=(rows * cols, ))
^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/numpy/ctypeslib.py", line 522, in as_array
obj = ctypes.cast(obj, p_arr_type).contents
^^^^^^^^^^^^^^^^^
ValueError: [address=0.0.0.0:37293, pid=88] NULL pointer access
```

I tried prebuilt `v0.2.77`, everything is ok.
Could you please provide some solutions for these issues? Thank you.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.