abetlen / abetlen/llama-cpp-python

Wrong chat format for llava 1.5

Open
#1,905 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
10.6k
Forks
1.4k
PR merge metrics
PR metrics pending

Description

# Prerequisites

Please answer the following questions for yourself before submitting an issue.

- [X] I am running the latest code. Development is very rapid so there are no tagged versions as of now.
- [X] I carefully followed the [README.md](https://github.com/abetlen/llama-cpp-python/blob/main/README.md).
- [X] I [searched using keywords relevant to my issue](https://docs.github.com/en/issues/tracking-your-work-with-issues/filtering-and-searching-issues-and-pull-requests) to make sure that I am creating a new issue that is not already open (or closed).
- [X] I reviewed the [Discussions](https://github.com/abetlen/llama-cpp-python/discussions), and have a new bug or useful enhancement to share.

# Current Behavior

I'm running llama-server with following command:
```
python3 -m llama_cpp.server --model models/mys/ggml_llava-v1.5-13b/ggml-model-q4_k.gguf --clip_model_path models/mys/ggml_llava-v1.5-13b/mmproj-model-f16.gguf --model_alias llava-v1.5-13b-q4_k --chat_format llava-1-5 --port 10322
```

(models downloaded from https://huggingface.co/mys/ggml_llava-v1.5-13b/tree/main)

When I call the server using openai python package:
```python
from openai import OpenAI

client = OpenAI(
base_url="http://localhost:10322/v1", # "http://:port"
api_key = "sk-no-key-required"
)

chat_completion = client.chat.completions.create(
model="models/mys/ggml_llava-v1.5-13b/ggml-model-q4_k.gguf",
messages=[
{"role": "user", "content": "Write a limerick about python exceptions"}
],
)
```

The server output this in console:
```
A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.USER: Write a limerick about python exceptionsUSER: ASSISTANT:
Llama.generate: 48 prefix-match hit, remaining 1 prompt tokens to eval
llama_perf_context_print: load time = 672844.44 ms
llama_perf_context_print: prompt eval time = 0.00 ms / 1 tokens ( 0.00 ms per token, inf tokens per second)
llama_perf_context_print: eval time = 0.00 ms / 32 runs ( 0.00 ms per token, inf tokens per second)
llama_perf_context_print: total time = 906.07 ms / 33 tokens
INFO: 127.0.0.1:40776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
```

As you can read, there is an additional unwanted `USER: ` at the end of the prompt.
I guess the chat format provided by `--chat_format llava-1-5` is not correct.

# Environment and Context

llama_cpp installed with `pip install llama-cpp-python[server]`
`print(llama_cpp.__version__)`: 0.3.6
`print(openai.__version__)`: 1.59.7

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.