abetlen / abetlen/llama-cpp-python

error when I using the Function Call

Abierto
#1,747 1 comentario 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Python
Estrellas
10.6k
Forks
1.4k
Métricas de merge de PR
Métricas de PR pendientes

Descripción

I am following this tutorial to use function call with qwen2:0.5b https://github.com/abetlen/llama-cpp-python/blob/main/examples/notebooks/Functions.ipynb

I used this command to start a serve inside container
```bash
python3 -m llama_cpp.server --model /app/vlm_weights/Qwen2-0___5B-Instruct-GGUF/qwen2-0_5b-instruct-fp16.gguf --n_gpu_layers -1
```

then I can successfully using following code
```python
import openai
import json
from pydantic import BaseModel
import instructor

client = openai.OpenAI(
api_key="sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", # can be anything
base_url="http://127.0.0.1:8000/v1", # NOTE: Replace with IP address and port of your llama-cpp-python server
)

# Enables `response_model`
client = instructor.patch(client=client)

class UserDetail(BaseModel):
name: str
age: int

user = client.chat.completions.create(
model="gpt-3.5-turbo",
response_model=UserDetail,
messages=[
{"role": "user", "content": "Extract Jason is 25 years old"},
],
)

assert isinstance(user, UserDetail)
assert user.name == "Jason"
assert user.age == 25

print(user)
```

with output
```bash
root@ubuntu:/app/inside_container/llama_python_demo# python3 client.py
name='Jason' age=25
```

but when I using Text Classification example with code
```python
import openai
import json
from pydantic import BaseModel
import enum

client = openai.OpenAI(
api_key="sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", # can be anything
base_url="http://127.0.0.1:8000/v1", # NOTE: Replace with IP address and port of your llama-cpp-python server
)

class Labels(str, enum.Enum):
"""Enumeration for single-label text classification."""

SPAM = "spam"
NOT_SPAM = "not_spam"

class SinglePrediction(BaseModel):
"""
Class for a single class label prediction.
"""

class_label: Labels

def classify(data: str) -> SinglePrediction:
"""Perform single-label classification on the input text."""
return client.chat.completions.create(
model="gpt-3.5-turbo-0613",
response_model=SinglePrediction,
messages=[
{
"role": "user",
"content": f"Classify the following text: {data}",
},
],
) # type: ignore

prediction = classify("Hello there I'm a Nigerian prince and I want to give you money")
assert prediction.class_label == Labels.SPAM
print(prediction)
```
with bug
```bash
root@ubuntu:/app/inside_container/llama_python_demo# python3 client.py
Traceback (most recent call last):
File "/app/inside_container/llama_python_demo/client.py", line 41, in
prediction = classify("Hello there I'm a Nigerian prince and I want to give you money")
File "/app/inside_container/llama_python_demo/client.py", line 29, in classify
return client.chat.completions.create(
File "/usr/local/lib/python3.10/dist-packages/openai/_utils/_utils.py", line 274, in wrapper
return func(*args, **kwargs)
TypeError: Completions.create() got an unexpected keyword argument 'response_model'
```

Can anyone help me look at it?

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.