deepseek-ai / deepseek-ai/DeepSeek-Coder

Why generate "GGGGG...." ,when the input string is longer than a certain length in GGUF model?

Open
#151 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
24.3k
Forks
2.9k
PR merge metrics
No merged PRs in 30d

Description

TheBloke/deepseek-coder-33B-instruct-GGUF
deepseek-coder-33b-instruct.Q6_K.gguf
when I use llama cpp python to load the model , the model generates endless "GGGG...." ,But It works normally when the input question less than about 1000-2000 words.

llm = Llama.from_pretrained(
repo_id=args.model_name_or_path,
# chat_format="llama-2",
# chat_format="alpaca",
filename="deepseek-coder-33b-instruct.Q6_K.gguf",
n_ctx=16000,
tokenizer=tokenizer,
n_gpu_layers=-1,


verbose=True
)

output = llm(
user_prompt
,
stream=True,
max_tokens=12000,
# max_new_tokens=4096,
# do_sample=False,
top_k=50,
top_p=0.95,
# num_return_sequences=1,
# eos_token_id=tokenizer.eos_token_id
# repeat_penalty=1,

)

for chunk in output:
delta = chunk['choices'][0]['text']
print(delta,end='')

bot_message+=delta

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.