abetlen / abetlen/llama-cpp-python

How get output from fine tuned llama3 model(trained with alpaca format dataset) in a json format ?

Abierto
#1,744 0 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Python
Estrellas
10.6k
Forks
1.4k
Métricas de merge de PR
Métricas de PR pendientes

Descripción

**Is your feature request related to a problem? Please describe.**
I have my own gguf file.
I was loading it like this :
```

from llama_cpp import Llama
llm = Llama(model_path=model_pth,
n_ctx=CONTEXT_SIZE,
chat_format="alpaca")
model_output = model(user_prompt,max_tokens=max_tokens,temperature=temperature,
top_p=top_p,echo=False,stop=stop)
print(model_ouput)
```

but the "model_ouput" is of string type, and its format is not consistant. Thus I cant extract desired output from this string using a common function.

**Describe the solution you'd like**
I want to know a way, where I can get output in a consistant structured format.

**Describe alternatives you've considered**
I've tried this, but it doesnt work

```
llm = Llama(model_path=model_pth,
n_ctx=CONTEXT_SIZE,
chat_format="alpaca")
llm.create_chat_completion(
messages=[
{
"role": "system",
"content": "You are a helpful assistant that outputs in JSON.",
},
{"role": "user", "content": "Who won the world series in 2020"},
],
response_format={
"type": "json_object",
},
temperature=0.7,
)

```
**Additional context**
I am new to this field, any suggestion is appriciated. Thanks !

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.