abetlen / abetlen/llama-cpp-python

How get output from fine tuned llama3 model(trained with alpaca format dataset) in a json format ?

オープン
#1,744 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
Python
スター
10.6k
フォーク
1.4k
PR マージ指標
PR 指標を取得中

説明

**Is your feature request related to a problem? Please describe.**
I have my own gguf file.
I was loading it like this :
```

from llama_cpp import Llama
llm = Llama(model_path=model_pth,
n_ctx=CONTEXT_SIZE,
chat_format="alpaca")
model_output = model(user_prompt,max_tokens=max_tokens,temperature=temperature,
top_p=top_p,echo=False,stop=stop)
print(model_ouput)
```

but the "model_ouput" is of string type, and its format is not consistant. Thus I cant extract desired output from this string using a common function.

**Describe the solution you'd like**
I want to know a way, where I can get output in a consistant structured format.

**Describe alternatives you've considered**
I've tried this, but it doesnt work

```
llm = Llama(model_path=model_pth,
n_ctx=CONTEXT_SIZE,
chat_format="alpaca")
llm.create_chat_completion(
messages=[
{
"role": "system",
"content": "You are a helpful assistant that outputs in JSON.",
},
{"role": "user", "content": "Who won the world series in 2020"},
],
response_format={
"type": "json_object",
},
temperature=0.7,
)

```
**Additional context**
I am new to this field, any suggestion is appriciated. Thanks !

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。