deepseek-ai / deepseek-ai/DeepSeek-Coder
Why generate "GGGGG...." ,when the input string is longer than a certain length in GGUF model?
- Dominant language
- Python
- Stars
- 24.3k
- Forks
- 2.9k
- PR merge metrics
- No merged PRs in 30d
Description
TheBloke/deepseek-coder-33B-instruct-GGUF
deepseek-coder-33b-instruct.Q6_K.gguf
when I use llama cpp python to load the model , the model generates endless "GGGG...." ,But It works normally when the input question less than about 1000-2000 words.
llm = Llama.from_pretrained(
repo_id=args.model_name_or_path,
# chat_format="llama-2",
# chat_format="alpaca",
filename="deepseek-coder-33b-instruct.Q6_K.gguf",
n_ctx=16000,
tokenizer=tokenizer,
n_gpu_layers=-1,
verbose=True
)
output = llm(
user_prompt
,
stream=True,
max_tokens=12000,
# max_new_tokens=4096,
# do_sample=False,
top_k=50,
top_p=0.95,
# num_return_sequences=1,
# eos_token_id=tokenizer.eos_token_id
# repeat_penalty=1,
)
for chunk in output:
delta = chunk['choices'][0]['text']
print(delta,end='')
bot_message+=delta
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.