abetlen / abetlen/llama-cpp-python

Unexpected behaviour when using temperature=0

Abierto
#890 6 comentarios 0 reacciones 0 asignados Ver en GitHub
bug
Lenguaje dominante
Python
Estrellas
10.6k
Forks
1.4k
Métricas de merge de PR
Métricas de PR pendientes

Descripción

### Problem description
Hi, I have been doing some basic testing in a notebook after finding some strange behavior in my code.
Basically two things happen when running a model with `temperature=0` for versions `>0.2.14`:
- The first response of the model is different from the following ones, and always contains a strange character (`▅`).
- The rest of the responses for the same prompt are the same, which is the expected behavior.
- If we then change the prompt to any other prompt, the first response will always be empty.
- This happens every time we change the prompt.

### Examples
It's easier to understand with examples, so I'll upload a couple of screenshots.
- First, using version 0.2.14. The behaviour is incorrect:
![image](https://github.com/abetlen/llama-cpp-python/assets/62963844/4801b139-98b6-456b-a6d4-31e3931768e3)
- Second, using version 0.2.13. The behaviour is correct:
![image](https://github.com/abetlen/llama-cpp-python/assets/62963844/b4679a31-b27f-46e5-89a4-5c814ddc2d93)

### Code
GitHub won't let me upload the notebook, so I'll just paste its cells:
```
from llama_cpp import Llama
```
```
def run_model(
llm: Llama, prompt: str, max_tokens: int = 3000, temperature: float = 0, **kwargs
) -> str:
output = llm(prompt, max_tokens=max_tokens, temperature=temperature, **kwargs)

if kwargs.get("stream", False):
return output
return output["choices"][0]["text"]
```
```
PROMPT_TEMPLATE = \
"""### System:
You are Stable Beluga 13B, an AI that follows instructions extremely well. Help as much as you can. Remember, be safe, and don't do anything illegal.

### User:
{user_query}

### Assistant:"""
```
```
LLAMA_MODEL_FILENAME = 'stablebeluga-13b.Q8_0.gguf'
N_GPU_LAYERS = 41
MODEL = Llama(
model_path=f'models/{LLAMA_MODEL_FILENAME}',
n_ctx=4096,
n_gpu_layers=N_GPU_LAYERS,
verbose=False
)
```

### Notes
- The model I'm using was downloaded from [here](https://huggingface.co/TheBloke/StableBeluga-13B-GGUF).
- I tried changing `n_gpu_layers`, but it didn't change the results.
- I switched between versions by running `!CMAKE_ARGS="-DLLAMA_CUBLAS=on" pip install llama-cpp-python --force-reinstall --upgrade --no-cache-dir`.

### Seed?
I've also been trying to generate deterministic responses with `temperature>0` by setting the random seed to a constant number (`seed` parameter), but it didn't work in version `0.2.14`.

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.