abetlen / abetlen/llama-cpp-python
Newlines in generation when using grammar
- Lenguaje dominante
- Python
- Estrellas
- 10.6k
- Forks
- 1.4k
- Métricas de merge de PR
- Métricas de PR pendientes
Descripción
Using `llama-cpp-python` w/ LangChain integration and [this PR](https://github.com/langchain-ai/langchain/pull/9712/files) to support grammars.
Test w/o `grammar_path`:
```
llm = LlamaCpp(
model_path="/Users/rlm/Desktop/Code/llama.cpp/llama-2-13b-chat.ggmlv3.q4_0.bin",
n_gpu_layers=n_gpu_layers,
n_batch=n_batch,
callback_manager=callback_manager,
verbose=True,
)
question = "What NFL team won the Super Bowl in the year Justin Bieber was born?"
llm(question)
```
The result is as expected.
---
Test w/ `grammar_path`:
```
llm = LlamaCpp(
model_path="/Users/rlm/Desktop/Code/llama.cpp/llama-2-13b-chat.ggmlv3.q4_0.bin",
n_gpu_layers=n_gpu_layers,
n_batch=n_batch,
f16_kv=True, # MUST set to True, otherwise you will run into problem after a couple of calls
callback_manager=callback_manager,
verbose=True,
grammar_path="/Users/rlm/Desktop/json.gbnf",
)
question = "Request: schedule a call at 8pm; Command:"
llm(question)
```
The result has a large number of newlines:
```
'{"schedule": {"date": "2018-09-14T20:00:00.000Z", "duration": 60}}\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'
```
Has anyone seen / resolved similar behavior?
Guía de contribución
Evaluación
Este issue todavía no se ha evaluado.