kvcache-ai / kvcache-ai/ktransformers
[Bug] DeekSeek-R1-0528 max_new_tokens limit?
- Dominant language
- Python
- Stars
- 19.5k
- Forks
- 1.6k
- Avg merge
- 19h 32m
- Merged PRs (30d)
- 27
Description
### Checklist
- [x] 1. I have searched related issues but cannot get the expected help.
- [x] 2. The bug has not been fixed in the latest version.
- [x] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
- [x] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
- [x] 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed.
### Describe the bug
DeepSeek-R1-0528's output is cut short when thinking sometimes.
### Reproduction
I start R1 0528 with:
```bash
python ktransformers/server/main.py \
--port 11434 \
--model_path /data/DeepSeek-R1 \
--model_name "DeepSeek-R1-0528:671b-q4_k_m" \
--gguf_path /data/DeepSeek-R1-0528/Q4_K_M \
--optimize_config_path /home/jesse/ktransformers/ktransformers/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml \
--cpu_infer 30 \
--temperature 0.6 \
--top_p 0.95 \
--cache_lens 65536 \
--chunk_size 256 \
--max_new_tokens 131072 \
--backend_type ktransformers
```
However, R1's output is still cut off sometimes when thinking. It certainly does not appear to have outputted 128k tokens.
### Environment
ubuntu 24.04.02
dual EPYC 9355
768gb ram
3090 24gb
latest ktransformers
using the ktransformers backend
Contributor guide
Research direction
Start by reproducing the cutoff with the command shown for ktransformers/server/main.py, including --max_new_tokens 131072 and the DeepSeek-V3-Chat.yaml optimization configuration. Trace how the server and ktransformers backend apply that limit, compare the requested and produced token counts, and establish a reproducible explanation for when the output stops.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100