abetlen / abetlen/llama-cpp-python
Unexpected behaviour when using temperature=0
- Vorherrschende Sprache
- Python
- Sterne
- 10.6k
- Forks
- 1.4k
- PR-Merge-Kennzahlen
- PR-Kennzahlen ausstehend
Beschreibung
### Problem description
Hi, I have been doing some basic testing in a notebook after finding some strange behavior in my code.
Basically two things happen when running a model with `temperature=0` for versions `>0.2.14`:
- The first response of the model is different from the following ones, and always contains a strange character (`▅`).
- The rest of the responses for the same prompt are the same, which is the expected behavior.
- If we then change the prompt to any other prompt, the first response will always be empty.
- This happens every time we change the prompt.
### Examples
It's easier to understand with examples, so I'll upload a couple of screenshots.
- First, using version 0.2.14. The behaviour is incorrect:

- Second, using version 0.2.13. The behaviour is correct:

### Code
GitHub won't let me upload the notebook, so I'll just paste its cells:
```
from llama_cpp import Llama
```
```
def run_model(
llm: Llama, prompt: str, max_tokens: int = 3000, temperature: float = 0, **kwargs
) -> str:
output = llm(prompt, max_tokens=max_tokens, temperature=temperature, **kwargs)
if kwargs.get("stream", False):
return output
return output["choices"][0]["text"]
```
```
PROMPT_TEMPLATE = \
"""### System:
You are Stable Beluga 13B, an AI that follows instructions extremely well. Help as much as you can. Remember, be safe, and don't do anything illegal.
### User:
{user_query}
### Assistant:"""
```
```
LLAMA_MODEL_FILENAME = 'stablebeluga-13b.Q8_0.gguf'
N_GPU_LAYERS = 41
MODEL = Llama(
model_path=f'models/{LLAMA_MODEL_FILENAME}',
n_ctx=4096,
n_gpu_layers=N_GPU_LAYERS,
verbose=False
)
```
### Notes
- The model I'm using was downloaded from [here](https://huggingface.co/TheBloke/StableBeluga-13B-GGUF).
- I tried changing `n_gpu_layers`, but it didn't change the results.
- I switched between versions by running `!CMAKE_ARGS="-DLLAMA_CUBLAS=on" pip install llama-cpp-python --force-reinstall --upgrade --no-cache-dir`.
### Seed?
I've also been trying to generate deterministic responses with `temperature>0` by setting the random seed to a constant number (`seed` parameter), but it didn't work in version `0.2.14`.
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.