kvcache-ai / kvcache-ai/ktransformers

[Bug] DeekSeek-R1-0528 max_new_tokens limit?

Open
#1,375 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
19.5k
Forks
1.6k
Avg merge
19h 32m
Merged PRs (30d)
27

Description

### Checklist

- [x] 1. I have searched related issues but cannot get the expected help.
- [x] 2. The bug has not been fixed in the latest version.
- [x] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
- [x] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
- [x] 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed.

### Describe the bug

DeepSeek-R1-0528's output is cut short when thinking sometimes.

### Reproduction

I start R1 0528 with:

```bash
python ktransformers/server/main.py \
--port 11434 \
--model_path /data/DeepSeek-R1 \
--model_name "DeepSeek-R1-0528:671b-q4_k_m" \
--gguf_path /data/DeepSeek-R1-0528/Q4_K_M \
--optimize_config_path /home/jesse/ktransformers/ktransformers/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml \
--cpu_infer 30 \
--temperature 0.6 \
--top_p 0.95 \
--cache_lens 65536 \
--chunk_size 256 \
--max_new_tokens 131072 \
--backend_type ktransformers
```

However, R1's output is still cut off sometimes when thinking. It certainly does not appear to have outputted 128k tokens.

### Environment

ubuntu 24.04.02
dual EPYC 9355
768gb ram
3090 24gb
latest ktransformers
using the ktransformers backend

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the cutoff with the command shown for ktransformers/server/main.py, including --max_new_tokens 131072 and the DeepSeek-V3-Chat.yaml optimization configuration. Trace how the server and ktransformers backend apply that limit, compare the requested and produced token counts, and establish a reproducible explanation for when the output stops.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.