abetlen / abetlen/llama-cpp-python
Fails when attempting to load llama.dll during Poetry final bootup.
- Langage dominant
- Python
- Étoiles
- 10.6k
- Forks
- 1.4k
- Métriques de merge des PR
- Métriques de PR en attente
Description
# Prerequisites
Please answer the following questions for yourself before submitting an issue.
- [ ] I am running the latest code. Development is very rapid so there are no tagged versions as of now.
- [x] I carefully followed the [README.md](https://github.com/abetlen/llama-cpp-python/blob/main/README.md).
- [x] I [searched using keywords relevant to my issue](https://docs.github.com/en/issues/tracking-your-work-with-issues/filtering-and-searching-issues-and-pull-requests) to make sure that I am creating a new issue that is not already open (or closed).
- [x] I reviewed the [Discussions](https://github.com/abetlen/llama-cpp-python/discussions), and have a new bug or useful enhancement to share.
# Expected Behavior
I am trying to run `poetry run python -m uvicorn private_gpt.main:app --reload --port 8001`. it should run without a hitch.
This has been installed as a lesser version (because that's what poetry was going to install) and inside a conda (miniconda to be exact) environment under Python 3.11.
# Current Behavior
When running it will move through a few steps until it attempts to load the llama dll. then it will fail. Changing the winmode around to different values and even inputting the exact location of the llama .dll doesn't fix it.
# Environment and Context
Please provide detailed information about your computer setup. This is important in case the issue is not reproducible except for under certain specific conditions.
```
Running in a virtual machine
KVM
Chipset Q35
Bios loaded
Processor Intel Xeon E312xx (Sandy Bridge) 2.00 GHz (2 cores 5 threads)
Installed RAM 11.0 GB
System type 64-bit operating system, x64-based processor
Edition Windows 10 Pro
Version 22H2
OS build 19045.3393
Experience Windows Feature Experience Pack 1000.19044.1000.0
```
* SDK version, e.g. for Linux:
```
Poetry (version 1.7.1)
Python 3.11.7
conda 23.7.3
llama 0.2.23+cu122
```
# Failure Information (for bugs)
```
Traceback (most recent call last):
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\site-packages\injector\__init__.py", line 798, in get
return self._context[key]
~~~~~~~~~~~~~^^^^^
KeyError:
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\site-packages\injector\__init__.py", line 798, in get
return self._context[key]
~~~~~~~~~~~~~^^^^^
KeyError:
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\site-packages\injector\__init__.py", line 798, in get
return self._context[key]
~~~~~~~~~~~~~^^^^^
KeyError:
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\site-packages\llama_cpp\llama_cpp.py", line 76, in _load_shared_library
return ctypes.CDLL(str(_lib_path), **cdll_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\ctypes\__init__.py", line 376, in __init__
self._handle = _dlopen(self._name, mode)
^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: Could not find module 'C:\ProgramData\miniconda3\envs\privateGPT\Lib\site-packages\llama_cpp\llama.dll' (or one of its dependencies). Try using the full path with constructor syntax.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\multiprocessing\process.py", line 314, in _bootstrap
self.run()
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\multiprocessing\process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\site-packages\uvicorn\_subprocess.py", line 76, in subprocess_started
target(sockets=sockets)
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\site-packages\uvicorn\server.py", line 61, in run
return asyncio.run(self.serve(sockets=sockets))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\asyncio\runners.py", line 190, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\asyncio\runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\asyncio\base_events.py", line 653, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\site-packages\uvicorn\server.py", line 68, in serve
config.load()
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\site-packages\uvicorn\config.py", line 467, in load
self.loaded_app = import_from_string(self.app)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\site-packages\uvicorn\importer.py", line 21, in import_from_string
module = importlib.import_module(module_str)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\importlib\__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 1204, in _gcd_import
File "", line 1176, in _find_and_load
File "", line 1147, in _find_and_load_unlocked
File "", line 690, in _load_unlocked
File "", line 940, in exec_module
File "", line 241, in _call_with_frames_removed
File "C:\Users\admin\Desktop\privateGPT\private_gpt\main.py", line 11, in
app = create_app(global_injector)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\admin\Desktop\privateGPT\private_gpt\launcher.py", line 50, in create_app
ui = root_injector.get(PrivateGptUi)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\site-packages\injector\__init__.py", line 91, in wrapper
return function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\site-packages\injector\__init__.py", line 974, in get
provider_instance = scope_instance.get(interface, binding.provider)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\site-packages\injector\__init__.py", line 91, in wrapper
return function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\site-packages\injector\__init__.py", line 800, in get
instance = self._get_instance(key, provider, self.injector)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\site-packages\injector\__init__.py", line 811, in _get_instance
return provider.get(injector)
^^^^^^^^^^^^^^^^^^^^^^
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\site-packages\injector\__init__.py", line 264, in get
return injector.create_object(self._cls)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\site-packages\injector\__init__.py", line 998, in create_object
self.call_with_injection(init, self_=instance, kwargs=additional_kwargs)
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\site-packages\injector\__init__.py", line 1031, in call_with_injection
dependencies = self.args_to_inject(
^^^^^^^^^^^^^^^^^^^^
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\site-packages\injector\__init__.py", line 91, in wrapper
return function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\site-packages\injector\__init__.py", line 1079, in args_to_inject
instance: Any = self.get(interface)
^^^^^^^^^^^^^^^^^^^
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\site-packages\injector\__init__.py", line 91, in wrapper
return function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\site-packages\injector\__init__.py", line 974, in get
provider_instance = scope_instance.get(interface, binding.provider)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\site-packages\injector\__init__.py", line 91, in wrapper
return function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\site-packages\injector\__init__.py", line 800, in get
instance = self._get_instance(key, provider, self.injector)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\site-packages\injector\__init__.py", line 811, in _get_instance
return provider.get(injector)
^^^^^^^^^^^^^^^^^^^^^^
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\site-packages\injector\__init__.py", line 264, in get
return injector.create_object(self._cls)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\site-packages\injector\__init__.py", line 998, in create_object
self.call_with_injection(init, self_=instance, kwargs=additional_kwargs)
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\site-packages\injector\__init__.py", line 1031, in call_with_injection
dependencies = self.args_to_inject(
^^^^^^^^^^^^^^^^^^^^
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\site-packages\injector\__init__.py", line 91, in wrapper
return function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\site-packages\injector\__init__.py", line 1079, in args_to_inject
instance: Any = self.get(interface)
^^^^^^^^^^^^^^^^^^^
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\site-packages\injector\__init__.py", line 91, in wrapper
return function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\site-packages\injector\__init__.py", line 974, in get
provider_instance = scope_instance.get(interface, binding.provider)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\site-packages\injector\__init__.py", line 91, in wrapper
return function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\site-packages\injector\__init__.py", line 800, in get
instance = self._get_instance(key, provider, self.injector)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\site-packages\injector\__init__.py", line 811, in _get_instance
return provider.get(injector)
^^^^^^^^^^^^^^^^^^^^^^
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\site-packages\injector\__init__.py", line 264, in get
return injector.create_object(self._cls)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\site-packages\injector\__init__.py", line 998, in create_object
self.call_with_injection(init, self_=instance, kwargs=additional_kwargs)
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\site-packages\injector\__init__.py", line 1040, in call_with_injection
return callable(*full_args, **dependencies)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\admin\Desktop\privateGPT\private_gpt\components\llm\llm_component.py", line 38, in __init__
self.llm = LlamaCPP(
^^^^^^^^^
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\site-packages\llama_index\llms\llama_cpp.py", line 102, in __init__
from llama_cpp import Llama
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\site-packages\llama_cpp\__init__.py", line 1, in
from .llama_cpp import *
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\site-packages\llama_cpp\llama_cpp.py", line 89, in
_lib = _load_shared_library(_lib_base_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ProgramData\miniconda3\envs\privateGPT\Lib\site-packages\llama_cpp\llama_cpp.py", line 78, in _load_shared_library
raise RuntimeError(f"Failed to load shared library '{_lib_path}': {e}")
RuntimeError: Failed to load shared library 'C:\ProgramData\miniconda3\envs\privateGPT\Lib\site-packages\llama_cpp\llama.dll': Could not find module 'C:\ProgramData\miniconda3\envs\privateGPT\Lib\site-packages\llama_cpp\llama.dll' (or one of its dependencies). Try using the full path with constructor syntax.
```
# Steps to Reproduce
1. Install everything according to [this guide](https://medium.com/data-and-beyond/run-privategpt-on-windows-bf64fe2a02b8)
2. When you get to the step about installing llama, install via pip instead (because I had to) and then change the version poetry was going to install to the version included in this report.
3. reach the step where you set up the PrivateGPT environment for running for the first time. (`poetry run python -m uvicorn private_gpt.main:app --reload --port 8001`)
I'm actually not entirely sure this is a llama issue, as I am extremely new to python and my coding skills leave a lot to be desired. I am submitting this here first because the bottom of the error stack (and sprinkled throughout) is that it cannot find llama.dll. As well as the fact it happens when I attempt to load same. But this is just a hunch on my part.
If there is anything I have forgotten or any additional information you need, please let me know.
Guide de contribution
Ouvrir le guide de contribution
Évaluation
Cette issue n'a pas encore été évaluée.