kvcache-ai / kvcache-ai/ktransformers
[Bug] KT 0.3.2版本 运行Kimi-K2-Instruct-UD-IQ2_XXS 能加载模型,进入对话框后报错。
- Dominant language
- Python
- Stars
- 19.5k
- Forks
- 1.6k
- Avg merge
- 19h 32m
- Merged PRs (30d)
- 27
Description
### 检查清单
- [ ] 1. 我已经搜索过相关问题,但未能获得预期的帮助
- [x] 2. 该问题在最新版本中尚未修复
- [ ] 3. 请注意,如果您提交的BUG相关 issue 缺少对应环境信息和最小可复现示例,我们将难以复现和定位问题,降低获得反馈的可能性
- [ ] 4. 如果您提出的不是bug而是问题,请在讨论区发起讨论 https://github.com/kvcache-ai/ktransformers/discussions。否则该 issue 将被关闭
- [ ] 5. 为方便社区交流,我将使用中文/英文或附上中文/英文翻译(如使用其他语言)。未附带翻译的非中文/英语内容可能会被关闭
### 问题描述
model_path: str = MODEL_PATH,
optimize_config_path: str = ktransformer_rules_dir+'DeepSeek-V3-Chat-serve.yaml',
gguf_path: str = GGUF_PATH, # | None = None,
max_new_tokens: int =1024,
cpu_infer: str = CPU_INFER ,#: int = Config().cpu_infer,
use_cuda_graph: bool = True , #False,
prompt_file : str | None = None,
mode: str = "normal",
force_think: bool = False,
chunk_size: int =12800
正常加载完成60层。
loading model.layers.60.self_attn.q_a_layernorm.weight to cuda
loading model.layers.60.self_attn.kv_a_layernorm.weight to cuda
loading model.layers.60.self_attn.kv_b_proj.weight to cuda:0
loading model.layers.60.input_layernorm.weight to cuda
loading model.layers.60.post_attention_layernorm.weight to cuda
loading model.norm.weight to cuda
Chat: 你好
显存占用:8G
WARNING:transformers_modules.tokenization_kimi:Calling super().encode with {'add_special_tokens': False}
报错如下:
File /opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py:1747, in Module._call_impl(self, *args, **kwargs)
1742 # If we don't have any hooks, we want to skip the rest of the logic in
1743 # this function, and just call forward.
1744 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
1745 or _global_backward_pre_hooks or _global_backward_hooks
1746 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1747 return forward_call(*args, **kwargs)
1749 result = None
1750 called_always_called_hooks = set()
**TypeError: flashinfer_attn.forward() got an unexpected keyword argument 'attention_mask'**
### 复现步骤
python ktransformers/server/main.py \
--port 10002 \
--model_path \
--gguf_path \
--optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml \
--max_new_tokens 1024 \
--cache_lens 32768 \
--chunk_size 256 \
--max_batch_size 4 \
--backend_type balance_serve \
反馈:
### 环境信息
ubuntu2204
X99
512 ddr4
2CPU 56线程
RTX3070 16G
docker 容器中跑(3.1.2.4都正常可以跑。
Contributor guide
Research direction
Start with the reproduction command for ktransformers/server/main.py and inspect the call path that reaches flashinfer_attn.forward(). Reproduce the failure in the stated Ubuntu, Docker, RTX 3070 environment, focusing on the unexpected attention_mask argument. Done means the Kimi-K2-Instruct-UD-IQ2_XXS model loads and entering a chat no longer raises this TypeError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100