abetlen / abetlen/llama-cpp-python
error when I using the Function Call
- 主要语言
- Python
- 星标
- 10.6k
- 派生
- 1.4k
- PR 合并指标
- PR 指标待抓取
描述
I am following this tutorial to use function call with qwen2:0.5b https://github.com/abetlen/llama-cpp-python/blob/main/examples/notebooks/Functions.ipynb
I used this command to start a serve inside container
```bash
python3 -m llama_cpp.server --model /app/vlm_weights/Qwen2-0___5B-Instruct-GGUF/qwen2-0_5b-instruct-fp16.gguf --n_gpu_layers -1
```
then I can successfully using following code
```python
import openai
import json
from pydantic import BaseModel
import instructor
client = openai.OpenAI(
api_key="sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", # can be anything
base_url="http://127.0.0.1:8000/v1", # NOTE: Replace with IP address and port of your llama-cpp-python server
)
# Enables `response_model`
client = instructor.patch(client=client)
class UserDetail(BaseModel):
name: str
age: int
user = client.chat.completions.create(
model="gpt-3.5-turbo",
response_model=UserDetail,
messages=[
{"role": "user", "content": "Extract Jason is 25 years old"},
],
)
assert isinstance(user, UserDetail)
assert user.name == "Jason"
assert user.age == 25
print(user)
```
with output
```bash
root@ubuntu:/app/inside_container/llama_python_demo# python3 client.py
name='Jason' age=25
```
but when I using Text Classification example with code
```python
import openai
import json
from pydantic import BaseModel
import enum
client = openai.OpenAI(
api_key="sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", # can be anything
base_url="http://127.0.0.1:8000/v1", # NOTE: Replace with IP address and port of your llama-cpp-python server
)
class Labels(str, enum.Enum):
"""Enumeration for single-label text classification."""
SPAM = "spam"
NOT_SPAM = "not_spam"
class SinglePrediction(BaseModel):
"""
Class for a single class label prediction.
"""
class_label: Labels
def classify(data: str) -> SinglePrediction:
"""Perform single-label classification on the input text."""
return client.chat.completions.create(
model="gpt-3.5-turbo-0613",
response_model=SinglePrediction,
messages=[
{
"role": "user",
"content": f"Classify the following text: {data}",
},
],
) # type: ignore
prediction = classify("Hello there I'm a Nigerian prince and I want to give you money")
assert prediction.class_label == Labels.SPAM
print(prediction)
```
with bug
```bash
root@ubuntu:/app/inside_container/llama_python_demo# python3 client.py
Traceback (most recent call last):
File "/app/inside_container/llama_python_demo/client.py", line 41, in
prediction = classify("Hello there I'm a Nigerian prince and I want to give you money")
File "/app/inside_container/llama_python_demo/client.py", line 29, in classify
return client.chat.completions.create(
File "/usr/local/lib/python3.10/dist-packages/openai/_utils/_utils.py", line 274, in wrapper
return func(*args, **kwargs)
TypeError: Completions.create() got an unexpected keyword argument 'response_model'
```
Can anyone help me look at it?
贡献指南
评估
这个 Issue 还没有评估数据。