abetlen / abetlen/llama-cpp-python

llama_get_logits_ith: invalid logits id -1, reason: no logits

Đang mở
#1,855 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Python
Star
10.6k
Fork
1.4k
Chỉ số merge pull request
Chỉ số pull request đang chờ

Mô tả

# Prerequisites

Please answer the following questions for yourself before submitting an issue.

- [x] I am running the latest code. Development is very rapid so there are no tagged versions as of now.
- [x] I carefully followed the [README.md](https://github.com/abetlen/llama-cpp-python/blob/main/README.md).
- [x] I [searched using keywords relevant to my issue](https://docs.github.com/en/issues/tracking-your-work-with-issues/filtering-and-searching-issues-and-pull-requests) to make sure that I am creating a new issue that is not already open (or closed).
- [x] I reviewed the [Discussions](https://github.com/abetlen/llama-cpp-python/discussions), and have a new bug or useful enhancement to share.

# Expected Behavior
Get a response from model
# Current Behavior
llama_get_logits_ith: invalid logits id -1, reason: no logits
[1] 58786 segmentation fault python ragPhiGguf.py --model_path Phi-3.5-mini-instruct-Q4_K_M.gguf --query
/opt/homebrew/Cellar/python@3.13/3.13.0_1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/multiprocessing/resource_tracker.py:276: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown: {'/loky-58786-dm8v3d6t'}
warnings.warn(

# Environment and Context

* Physical (or virtual) hardware you are using, e.g. for Linux:
arm64 m2 pro

* Operating System, e.g. for Linux:
mac

* SDK version, e.g. for Linux:
Python 3.13.0

GNU Make 3.81
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for i386-apple-darwin11.3.0

g++ --version
Apple clang version 16.0.0 (clang-1600.0.26.4)
Target: arm64-apple-darwin24.1.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

# Failure Information (for bugs)

segmentation fault

# Steps to Reproduce

i was trying to build a RAG script using `Phi-3.5-mini-instruct-Q4_K_M.gguf` and i am using default chat template

```
Available chat formats from metadata: chat_template.default
Using gguf chat template: {% for message in messages %}{% if message['role'] == 'system' and message['content'] %}{{'<|system|>
' + message['content'] + '<|end|>
'}}{% elif message['role'] == 'user' %}{{'<|user|>
' + message['content'] + '<|end|>
'}}{% elif message['role'] == 'assistant' %}{{'<|assistant|>
' + message['content'] + '<|end|>
'}}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ '<|assistant|>
' }}{% else %}{{ eos_token }}{% endif %}
```

```
def generate_response(
self, query: str, context_docs: List[Dict], max_tokens: int = 1000
) -> str:
"""
Generate response using retrieved documents as context, formatted in GGUF chat template.
Args:
query (str): User query
context_docs (List[Dict]): Retrieved context documents
max_tokens (int): Maximum tokens to generate
Returns:
Generated response
"""
# Construct context from documents
context_texts = []
for doc in context_docs:
context_texts.append(
f"Article ID: {doc.get('article_id', 'N/A')
}\n{doc.get('text', '')}"
)
context = "\n\n".join(context_texts)
# Ensure context fits within the context window
context = self.truncate_text(context)

# Construct GGUF chat template prompt
prompt = (
"{% for message in messages %}"
"{% if message['role'] == 'system' and message['content'] %}"
"{{'<|system|>\n' + message['content'] + '<|end|>\n'}}"
"{% elif message['role'] == 'user' %}"
"{{'<|user|>\n' + message['content'] + '<|end|>\n'}}"
"{% elif message['role'] == 'assistant' %}"
"{{'<|assistant|>\n' + message['content'] + '<|end|>\n'}}"
"{% endif %}{% endfor %}"
"{% if add_generation_prompt %}"
"{{ '<|assistant|>\n' }}"
"{% else %}{{ eos_token }}{% endif %}"
)

template = Template(prompt)

# Replace placeholders with the actual chat history
messages = [
{"role": "system", "content": f"Context:\n{context}"},
{"role": "user", "content": query},
]
formatted_prompt = template.render(
messages=messages, add_generation_prompt=True, eos_token="<|endoftext|>"
)

# Generate response
try:
response = self.llm(
formatted_prompt,
max_tokens=max_tokens,
stop=["<|end|>"],
echo=False,
)
return response["choices"][0]["text"].strip()
except Exception as e:
self.logger.error(f"Response generation error: {e}")
return f"Error generating response: {str(e)}"
```

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.