abetlen / abetlen/llama-cpp-python
Newlines in generation when using grammar
- Ngôn ngữ chính
- Python
- Star
- 10.6k
- Fork
- 1.4k
- Chỉ số merge pull request
- Chỉ số pull request đang chờ
Mô tả
Using `llama-cpp-python` w/ LangChain integration and [this PR](https://github.com/langchain-ai/langchain/pull/9712/files) to support grammars.
Test w/o `grammar_path`:
```
llm = LlamaCpp(
model_path="/Users/rlm/Desktop/Code/llama.cpp/llama-2-13b-chat.ggmlv3.q4_0.bin",
n_gpu_layers=n_gpu_layers,
n_batch=n_batch,
callback_manager=callback_manager,
verbose=True,
)
question = "What NFL team won the Super Bowl in the year Justin Bieber was born?"
llm(question)
```
The result is as expected.
---
Test w/ `grammar_path`:
```
llm = LlamaCpp(
model_path="/Users/rlm/Desktop/Code/llama.cpp/llama-2-13b-chat.ggmlv3.q4_0.bin",
n_gpu_layers=n_gpu_layers,
n_batch=n_batch,
f16_kv=True, # MUST set to True, otherwise you will run into problem after a couple of calls
callback_manager=callback_manager,
verbose=True,
grammar_path="/Users/rlm/Desktop/json.gbnf",
)
question = "Request: schedule a call at 8pm; Command:"
llm(question)
```
The result has a large number of newlines:
```
'{"schedule": {"date": "2018-09-14T20:00:00.000Z", "duration": 60}}\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n'
```
Has anyone seen / resolved similar behavior?
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.