abetlen / abetlen/llama-cpp-python

Unexpected behaviour when using temperature=0

Aberta
#890 6 comentários 0 reações 0 responsáveis Ver no GitHub
bug
Linguagem predominante
Python
Estrelas
10.6k
Forks
1.4k
Métricas de merge de PRs
Métricas de PR pendentes

Descrição

### Problem description
Hi, I have been doing some basic testing in a notebook after finding some strange behavior in my code.
Basically two things happen when running a model with `temperature=0` for versions `>0.2.14`:
- The first response of the model is different from the following ones, and always contains a strange character (`▅`).
- The rest of the responses for the same prompt are the same, which is the expected behavior.
- If we then change the prompt to any other prompt, the first response will always be empty.
- This happens every time we change the prompt.

### Examples
It's easier to understand with examples, so I'll upload a couple of screenshots.
- First, using version 0.2.14. The behaviour is incorrect:
![image](https://github.com/abetlen/llama-cpp-python/assets/62963844/4801b139-98b6-456b-a6d4-31e3931768e3)
- Second, using version 0.2.13. The behaviour is correct:
![image](https://github.com/abetlen/llama-cpp-python/assets/62963844/b4679a31-b27f-46e5-89a4-5c814ddc2d93)

### Code
GitHub won't let me upload the notebook, so I'll just paste its cells:
```
from llama_cpp import Llama
```
```
def run_model(
llm: Llama, prompt: str, max_tokens: int = 3000, temperature: float = 0, **kwargs
) -> str:
output = llm(prompt, max_tokens=max_tokens, temperature=temperature, **kwargs)

if kwargs.get("stream", False):
return output
return output["choices"][0]["text"]
```
```
PROMPT_TEMPLATE = \
"""### System:
You are Stable Beluga 13B, an AI that follows instructions extremely well. Help as much as you can. Remember, be safe, and don't do anything illegal.

### User:
{user_query}

### Assistant:"""
```
```
LLAMA_MODEL_FILENAME = 'stablebeluga-13b.Q8_0.gguf'
N_GPU_LAYERS = 41
MODEL = Llama(
model_path=f'models/{LLAMA_MODEL_FILENAME}',
n_ctx=4096,
n_gpu_layers=N_GPU_LAYERS,
verbose=False
)
```

### Notes
- The model I'm using was downloaded from [here](https://huggingface.co/TheBloke/StableBeluga-13B-GGUF).
- I tried changing `n_gpu_layers`, but it didn't change the results.
- I switched between versions by running `!CMAKE_ARGS="-DLLAMA_CUBLAS=on" pip install llama-cpp-python --force-reinstall --upgrade --no-cache-dir`.

### Seed?
I've also been trying to generate deterministic responses with `temperature>0` by setting the random seed to a constant number (`seed` parameter), but it didn't work in version `0.2.14`.

Guia de contribuição

Abrir o guia de contribuição

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.