kvcache-ai / kvcache-ai/ktransformers

[Bug] AssertionError: [Smallthinker config sanitizer] rope_layout must have the same length as num_hidden_layers

Open
#1,462 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
19.5k
Forks
1.6k
Avg merge
19h 32m
Merged PRs (30d)
27

Description

### Checklist

- [x] 1. I have searched related issues but cannot get the expected help.
- [ ] 2. The bug has not been fixed in the latest version.
- [ ] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
- [ ] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
- [ ] 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed.

### Describe the bug

python ktransformers/server/main.py \
--model_path /home/dhr/projects/dhr/kf/ktransformers/deepseek_v3 \
--gguf_path /home/dhr/projects/dhr/kf/ktransformers/deepseek_v3_q4_gguf \
--cpu_infer 15 \
--optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml \
--port 10002 \
--chunk_size 256 \
--max_new_tokens 1024 \
--max_batch_size 4 \
--port 10002 \
--cache_lens 3000 \
--backend_type balance_serve

是我模型下错了吗?但是我使用local_chat的时候是可行的

python3 -m ktransformers.local_chat \
--model_path "/home/dhr/projects/dhr/kf/ktransformers/deepseek_v3" \
--gguf_path "/home/dhr/projects/dhr/kf/ktransformers/deepseek_v3_q4_gguf"

### Reproduction

deepseek-v3

### Environment

按照install.md中的步骤创建的环境

Contributor guide

Open the contributing guide

Research direction

Start with ktransformers/server/main.py and compare its DeepSeek-V3 startup path with python3 -m ktransformers.local_chat, using the commands in the report. Review ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml and the install.md environment steps. Done means the server reproduces no rope_layout assertion, or the missing compatibility details are documented with a minimal reproduction.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.