abetlen / abetlen/llama-cpp-python

Problem with ChatCompletionMessage when using cuBLAS and Mistral.

Offen
#1,144 4 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
bug
Vorherrschende Sprache
Python
Sterne
10.6k
Forks
1.4k
PR-Merge-Kennzahlen
PR-Kennzahlen ausstehend

Beschreibung

# Expected Behavior

I expected the LM to output something, specifically output something into a database then output the result of the database entry, basically just a chat with a database in the middle. I was trying to use ChatCompletionMessage to pre-process messages so they'd go into a database neatly (following instructions from someone else). This code works when llama-cpp-python is compiled without cuBLAS. But with cuBLAS it loads the model to GPU memory like it should, it only crashes when it tries to generate a response to my input, or possibly when it gets the input. It might be Mistral specific, but I had no issue using the same Mistral model when llama-cpp-python was compiled without cuBLAS which is why I think it's a bug.

# Current Behavior

It failed to run giving me an undefined error related to jinja2 which my AI tells me is related to storing data in XML and HTML files and stuff, I'm assuming it's handling the instruction format or something, but I'm too noob to really know. Here is the error:

```
Traceback (most recent call last):
File "C:\LLM\chat.py", line 97, in
handle_message(message_content, author_name)
File "C:\LLM\chat.py", line 78, in handle_message
for token in completion:
File "C:\LLM\llama.py", line 68, in chat_completion
completion = self.model.create_chat_completion(chat_messages, stream=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\LLM\envs\Lib\site-packages\llama_cpp\llama.py", line 1523, in create_chat_completion
return handler(
^^^^^^^^
File "C:\LLM\envs\Lib\site-packages\llama_cpp\llama_chat_format.py", line 322, in chat_completion_handler
result = chat_formatter(
^^^^^^^^^^^^^^^
File "C:\LLM\envs\Lib\site-packages\llama_cpp\llama_chat_format.py", line 195, in __call__
prompt = self._environment.render(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\LLM\envs\Lib\site-packages\jinja2\environment.py", line 1301, in render
self.environment.handle_exception()
File "C:\LLM\envs\Lib\site-packages\jinja2\environment.py", line 936, in handle_exception
raise rewrite_traceback_stack(source=source)
File "", line 1, in top-level template code
File "C:\LLM\envs\Lib\site-packages\jinja2\utils.py", line 83, in from_obj
if hasattr(obj, "jinja_pass_arg"):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
jinja2.exceptions.UndefinedError: 'raise_exception' is undefined
```

The line it traces back to is just a very simple chat prompt at the end of my script I'm using for testing and shouldn't be relevant, it looks like this:
```
try:
while True:
message_content = input("Enter your message: ")
author_name = "User"
handle_message(message_content, author_name)
finally:
# Delete the chat from the database
DB.delete_all_chats()
```
The problem part seems to be here:
`completion = self.model.create_chat_completion(chat_messages, stream=True)`

# Environment and Context

Conda environment using python 3.12, llama-cpp-python is installed like this:

$env:FORCE_CMAKE='1'; $env:CMAKE_ARGS='-DLLAMA_CUBLAS=on'
pip3 install llama-cpp-python --no-cache-dir --verbose

PyTorch like this:
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

I also have these packages installed using pip in the environment:
python-dotenv
pymongo
pydantic<2.0

* Physical (or virtual) hardware you are using:

RTX 4090, i9-13900K, lots of memory.

* Operating System:

Windows 11

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.