abetlen / abetlen/llama-cpp-python

cohere multilanguage aya model seem can not use response_format feature of llama-cpp-python

Aperta
#1,483 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Python
Stelle
10.6k
Fork
1.4k
Metriche di merge delle PR
Metriche PR in attesa

Descrizione

Use gguf of https://huggingface.co/CohereForAI/aya-23-8B
from https://huggingface.co/bartowski/aya-23-8B-GGUF

```python
import llama_cpp
llm = llama_cpp.Llama.from_pretrained(
repo_id="bartowski/aya-23-8B-GGUF",
filename="aya-23-8B-Q4_K_M.gguf",
#tokenizer=llama_cpp.llama_tokenizer.LlamaHFTokenizer.from_pretrained("CohereForAI/aya-23-8B"),
verbose=False,
n_gpu_layers = -1,
n_ctx = 3060 * 3
)

prompt = '''
将下面的json内容翻译成中文,并保留相应的json格式:
{'problem_description': "Two space agencies, Galactic Explorations and Interstellar Missions, are discussing the potential of Planet X-31 for human colonization. Galactic Explorations claims that Planet X-31 is an ideal candidate due to its Earth-like atmosphere and abundant water resources. Interstellar Missions, however, argues that Planet X-31 is not suitable for colonization because of its high levels of radiation, which they claim would make it impossible for humans to survive there. Galactic Explorations counters this argument by stating that humans could develop technology to shield themselves from radiation in the future. Which statement best describes the fallacy in Galactic Explorations' argument?", 'additional_problem_info': "A) The fallacy is that Galactic Explorations assumes humans can develop technology to shield themselves from radiation without any evidence. \nB) The fallacy is that Interstellar Missions is incorrect about the high levels of radiation on Planet X-31. \nC) The fallacy is that Galactic Explorations believes Planet X-31 is the only planet suitable for human colonization. \nD) The fallacy is that Interstellar Missions doesn't believe in the potential of human technological advancements.", 'chain_of_thought': "Galactic Explorations' argument assumes that humans will be able to develop technology to shield themselves from radiation in the future. However, there is no evidence presented in the problem description to support this claim. Therefore, their argument contains a fallacy.", 'correct_solution': 'A) The fallacy is that Galactic Explorations assumes humans can develop technology to shield themselves from radiation without any evidence.'}
'''

from IPython.display import clear_output
messages = [
{
"role": "user",
"content": prompt
}
]
response = llm.create_chat_completion(
messages=messages,
response_format={
"type": "json_object",
"schema": {
"type": "object",
"properties": {
"problem_description": {"type": "string"},
"additional_problem_info": {"type": "string"},
"chain_of_thought": {"type": "string"},
"correct_solution": {"type": "string"},
},
"required": ["problem_description", "additional_problem_info", "chain_of_thought", "correct_solution"],
}
},
stream=True,
)

req = ""
for chunk in response:
delta = chunk["choices"][0]["delta"]
if "content" not in delta:
continue
#print(delta["content"], end="", flush=True)
req += delta["content"]
clear_output(wait = True)
print(req)
```

when I run this, python kernel died.
Can someone help me ?😊

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.