abetlen / abetlen/llama-cpp-python

Gibberish responses with Llama-2-13B

未關閉
#596 9 則留言 2 個 reaction 已指派 0 人 在 GitHub 檢視
model quality
主要語言
Python
星號
10.6k
分支
1.4k
PR 合併指標
PR 指標待擷取

描述

I am testing this nice python wrapper for llama.cpp. But the model's responses don't make much sense.

```
llm = Llama(model_path="./models/llama-2-13b.ggmlv3.q4_0.bin", n_gpu_layers=35, n_ctx=2048)
output = llm("What is the capital of Germany? Answer only with the name of the capital.", echo=True, temperature=0, max_tokens=512)
```

Gives the following output:

```
What is the capital of Germany? Answer only with the name of the capital.
What is the capital of France? Answer only with the name of the capital.
What is the capital of Italy? Answer only with the name of the capital.
What is the capital of Spain? Answer only with the name of the capital.
What is the capital of Portugal? Answer only with the name of the capital.
....
```

I wonder if the default hyperparameters of llama-cpp-python significantly differ from llama.cpp?

Either way this kind of response shouldn't be the case. I tested similar prompts and the model easily breaks down like above.

Needless to say the responses are as expected from using llama.cpp itself.

Am I missing something?

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。