abetlen / abetlen/llama-cpp-python

logprobs aren't returned if sampler will reach EOS

未關閉
#1,528 8 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
bug
主要語言
Python
星號
10.6k
分支
1.4k
PR 合併指標
PR 指標待擷取

描述

# Prerequisites

```python
llm = Llama(
model_path="/home/axyo/dev/LLM/models/Meta-Llama-3-8B-Instruct-GGUF-v2/Meta-Llama-3-8B-Instruct-v2.Q5_0.gguf",
n_gpu_layers=-1,
seed=8,
n_ctx=4096,
logits_all=True,
kv_overrides={"tokenizer.ggml.eos_token_id": 128002},
)

prompt = """<|start_header_id|>user<|end_header_id|>

What is a dog?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

A dog, also known as Canis lupus familiaris, is a domesticated mammal that belongs to the family Canidae. Dogs are closely related to wolves and share many physical and behavioral characteristics with them.

Dogs are typically characterized by their:

1. Physical appearance: Dogs come in a wide range of sizes, shapes, and coat types, from the tiny Chihuahua to the massive Great Dane. They have a furry coat, a wagging tail, and a distinctive snout.
2. Behavior: Dogs are social animals that thrive on interaction with their human family and other dogs. They are known for their loyalty, affection, and ability to form strong bonds with their owners.
3. Diet: Dogs are omnivores, which means they eat both plants and animals. They are often fed a diet of commercial dog food, which is formulated to meet their nutritional needs.
4. Communication: Dogs communicate primarily through body language and vocalizations, such as barking, whining, and growling.
5. Intelligence: Dogs are highly intelligent animals that are capable of learning a wide range of tasks and commands.

Dogs have been domesticated for thousands of years and have been bred for various purposes, such as:

* Companionship: Many dogs are kept as pets and are valued for their companionship and affection.
* Hunting: Some breeds of dogs, such as retrievers and pointers, are bred for hunting and retrieving game.
* Guarding: Some breeds, such as guard dogs and watchdogs, are bred for their protective instincts and ability to defend their territory.
* Assistance: Some dogs are trained as service animals to assist people with disabilities, such as guide dogs for the blind or hearing dogs for the deaf.

Overall, dogs are beloved animals that bring joy, companionship, and love to many people around the world

Would you like to know more about a specific aspect of dogs, such as their behavior, health, or training?"""

output = llm(
prompt,
echo=False,
logprobs=100,
max_tokens=1,

repeat_penalty=1.0, # disable penalties

top_k=1,
#top_p=1.0,
#min_p=-10000000000.0,
temperature=0,
)
print(output)
```

# Expected Behavior

logprobs!!

# Current Behavior

```
{'id': 'cmpl-c480f5df-6a62-4a39-9d21-d3058260523b', 'object': 'text_completion', 'created': 1718328460, 'model': '/home/axyo/dev/LLM/models/Meta-Llama-3-8B-Instruct-GGUF-v2/Meta-Llama-3-8B-Instruct-v2.Q5_0.gguf', 'choices': [{'text': '', 'index': 0, 'logprobs': {'tokens': [], 'text_offset': [], 'token_logprobs': [], 'top_logprobs': []}, 'finish_reason': 'stop'}], 'usage': {'prompt_tokens': 430, 'completion_tokens': 0, 'total_tokens': 430}}
```

# Environment and Context

llama_cpp_python-0.2.78

CUDA 12.4

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。