kvcache-ai / kvcache-ai/ktransformers

[Bug] Deepseek R1 - 0528 Function Calling - Template Bug

Open
#1,352 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
19.5k
Forks
1.6k
Avg merge
19h 32m
Merged PRs (30d)
27

Description

### Checklist

- [x] 1. I have searched related issues but cannot get the expected help.
- [x] 2. The bug has not been fixed in the latest version.
- [x] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
- [ ] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
- [ ] 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed.

### Describe the bug

Hi,

Thanks for the hardwork in fixing the new GGUF format for DeepSeek R1 0528. I am however running in to an issue when trying to do function calling. I am using the latest template from the official DeepSeek team. However, when trying to do a simple function call, ktransformers raises the following error:

```
File "/home/myfrienderic/miniconda3/envs/ktrans3.0-fix/lib/python3.11/site-packages/ktransformers/server/api/openai/endpoints/chat.py", line 436, in chat_completion
async for res in interface.inference(input_message, id, create.temperature, create.top_p, create.max_tokens, create.max_completion_tokens):
File "/home/myfrienderic/miniconda3/envs/ktrans3.0-fix/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 427, in inference
input_ids = self.format_and_tokenize_input_ids(thread_id, local_messages)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myfrienderic/miniconda3/envs/ktrans3.0-fix/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 413, in format_and_tokenize_input_ids
input_str: str = self.tokenizer.apply_chat_template(messages,tokenize=False,add_generation_prompt=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myfrienderic/miniconda3/envs/ktrans3.0-fix/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 1695, in apply_chat_template
rendered_chat = compiled_template.render(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myfrienderic/miniconda3/envs/ktrans3.0-fix/lib/python3.11/site-packages/jinja2/environment.py", line 1304, in render
self.environment.handle_exception()
File "/home/myfrienderic/miniconda3/envs/ktrans3.0-fix/lib/python3.11/site-packages/jinja2/environment.py", line 939, in handle_exception
raise rewrite_traceback_stack(source=source)
File "", line 3, in top-level template code
TypeError: argument of type 'NoneType' is not iterable
```

Has anyone else run into this or has any tips on how best to debug this?

Thanks!

### Reproduction

python ktransformers/server/main.py --model_path /mnt/home_extend/models/data/DeepSeek-R1-0528 --gguf_path /mnt/home_extend/models/unsloth_DeepSeek-R1-0528-GGUF/Q4_K_M --model_name DeepSeek-R1 --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml --cpu_infer 44 --max_new_tokens 30000 --cache_lens 120000 --chunk_size 512 --max_batch_size 4 --backend_type balance_serve --port 8088 --host 10.0.0.5

### Environment

Conda

Contributor guide

Open the contributing guide

Research direction

Start with balance_serve.py around format_and_tokenize_input_ids at line 413 and inference at line 427, then trace the chat_completion call in chat.py at line 436. Reproduce with the provided DeepSeek-R1 command and inspect the tokenizer's apply_chat_template failure. Done means function-calling input renders without the NoneType error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.