abetlen / abetlen/llama-cpp-python

Unexpected behaviour when using temperature=0

Đang mở
#890 6 bình luận 0 reaction 0 người được giao Xem trên GitHub
bug
Ngôn ngữ chính
Python
Star
10.6k
Fork
1.4k
Chỉ số merge pull request
Chỉ số pull request đang chờ

Mô tả

### Problem description
Hi, I have been doing some basic testing in a notebook after finding some strange behavior in my code.
Basically two things happen when running a model with `temperature=0` for versions `>0.2.14`:
- The first response of the model is different from the following ones, and always contains a strange character (`▅`).
- The rest of the responses for the same prompt are the same, which is the expected behavior.
- If we then change the prompt to any other prompt, the first response will always be empty.
- This happens every time we change the prompt.

### Examples
It's easier to understand with examples, so I'll upload a couple of screenshots.
- First, using version 0.2.14. The behaviour is incorrect:
![image](https://github.com/abetlen/llama-cpp-python/assets/62963844/4801b139-98b6-456b-a6d4-31e3931768e3)
- Second, using version 0.2.13. The behaviour is correct:
![image](https://github.com/abetlen/llama-cpp-python/assets/62963844/b4679a31-b27f-46e5-89a4-5c814ddc2d93)

### Code
GitHub won't let me upload the notebook, so I'll just paste its cells:
```
from llama_cpp import Llama
```
```
def run_model(
llm: Llama, prompt: str, max_tokens: int = 3000, temperature: float = 0, **kwargs
) -> str:
output = llm(prompt, max_tokens=max_tokens, temperature=temperature, **kwargs)

if kwargs.get("stream", False):
return output
return output["choices"][0]["text"]
```
```
PROMPT_TEMPLATE = \
"""### System:
You are Stable Beluga 13B, an AI that follows instructions extremely well. Help as much as you can. Remember, be safe, and don't do anything illegal.

### User:
{user_query}

### Assistant:"""
```
```
LLAMA_MODEL_FILENAME = 'stablebeluga-13b.Q8_0.gguf'
N_GPU_LAYERS = 41
MODEL = Llama(
model_path=f'models/{LLAMA_MODEL_FILENAME}',
n_ctx=4096,
n_gpu_layers=N_GPU_LAYERS,
verbose=False
)
```

### Notes
- The model I'm using was downloaded from [here](https://huggingface.co/TheBloke/StableBeluga-13B-GGUF).
- I tried changing `n_gpu_layers`, but it didn't change the results.
- I switched between versions by running `!CMAKE_ARGS="-DLLAMA_CUBLAS=on" pip install llama-cpp-python --force-reinstall --upgrade --no-cache-dir`.

### Seed?
I've also been trying to generate deterministic responses with `temperature>0` by setting the random seed to a constant number (`seed` parameter), but it didn't work in version `0.2.14`.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.