abetlen / abetlen/llama-cpp-python

Compatibility with LibreChat: According to Librechat, the server is not serving a OpenAI complaint API for mistral models with llama.cpp chat format `mistral-instruct`

未關閉
#1,174 1 則留言 1 個 reaction 已指派 0 人 在 GitHub 檢視
主要語言
Python
星號
10.6k
分支
1.4k
PR 合併指標
PR 指標待擷取

描述

# Prerequisites

Please answer the following questions for yourself before submitting an issue.

- [TRUE ] I am running the latest code. Development is very rapid so there are no tagged versions as of now.
- [ TRUE] I carefully followed the [README.md](https://github.com/abetlen/llama-cpp-python/blob/main/README.md).
- [ TRUE] I [searched using keywords relevant to my issue](https://docs.github.com/en/issues/tracking-your-work-with-issues/filtering-and-searching-issues-and-pull-requests) to make sure that I am creating a new issue that is not already open (or closed).
- [ TRUE] I reviewed the [Discussions](https://github.com/abetlen/llama-cpp-python/discussions), and have a new bug or useful enhancement to share.

llama-cpp-python has a OpenAI compatible server

I am serving a model as:
```
GGML_SYCL_DEVICE=0 python3 -m llama_cpp.server --model mistral-7b-instruct-v0.2.Q8_0.gguf --chat_format mistral-instruct --host 0.0.0.0 --port 7836 --n_ctx 16192 --n_gpu_layers 35

```

Output from llama-cpp-python:
```
INFO: Started server process [12022]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:7836 (Press CTRL+C to quit)
INFO: 192.168.3.114:51186 - "GET /v1/models HTTP/1.1" 200 OK # Librechat is requesting
INFO: 192.168.3.114:51198 - "POST /v1 HTTP/1.1" 404 Not Found # Librechat is requesting
```

I have checked ```http://192.168.3.113:7836/docs``` works

This is my librechat output when I try to chat with my model

```
Something went wrong. Here's the specific error message we encountered: Failed to send message. HTTP 404 - {"detail":"Not Found"}
```

This is my librechat.yaml configuration

```
# Configuration version (required)
version: 1.0.1

# Cache settings: Set to true to enable caching
cache: true

# Definition of custom endpoints
endpoints:
custom:

- name: "llama"
apiKey: "sk-1234"
baseURL: "http://192.168.3.113:7836/v1" # I have tried with just "http://192.168.3.113:7836" also, librechat again hits invalid endpoint
iconURL: ""
models:
default: ["mistral-7b-v0.1", "mistral-7b-instruct-v0.2"] # I was loading multiple models first, but I was trying to diagnose the error with 1 model only
fetch: true
titleConvo: true
titleModel: "mistral-7b-v0.1"
titleMethod: "completion"
summarize: true
summaryModel: "mistral-7b-v0.1"
forcePrompt: true
modelDisplayLabel : "llama"
```

Instead of the yaml file, I have also tried to modify the OPENAI_REVERSE_PROXY in `.env` file. (the old way of librechat endpoints)

**Possibilities**:
1. llama-cpp-python is not serving a OpenAI compatible server or the api endpoints are incomplete or not redirecting properly
2. I am missing some configuration in Librechat, since chat format is `--chat_format mistral-instruct`
3. I am missing some configuration for llama-cpp-python with chat format is `--chat_format mistral-instruct`

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。