kvcache-ai / kvcache-ai/ktransformers
[Bug] Deepseek R1 - 0528 Function Calling - Template Bug
- Dominant language
- Python
- Stars
- 19.5k
- Forks
- 1.6k
- Avg merge
- 19h 32m
- Merged PRs (30d)
- 27
Description
### Checklist
- [x] 1. I have searched related issues but cannot get the expected help.
- [x] 2. The bug has not been fixed in the latest version.
- [x] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
- [ ] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
- [ ] 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed.
### Describe the bug
Hi,
Thanks for the hardwork in fixing the new GGUF format for DeepSeek R1 0528. I am however running in to an issue when trying to do function calling. I am using the latest template from the official DeepSeek team. However, when trying to do a simple function call, ktransformers raises the following error:
```
File "/home/myfrienderic/miniconda3/envs/ktrans3.0-fix/lib/python3.11/site-packages/ktransformers/server/api/openai/endpoints/chat.py", line 436, in chat_completion
async for res in interface.inference(input_message, id, create.temperature, create.top_p, create.max_tokens, create.max_completion_tokens):
File "/home/myfrienderic/miniconda3/envs/ktrans3.0-fix/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 427, in inference
input_ids = self.format_and_tokenize_input_ids(thread_id, local_messages)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myfrienderic/miniconda3/envs/ktrans3.0-fix/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 413, in format_and_tokenize_input_ids
input_str: str = self.tokenizer.apply_chat_template(messages,tokenize=False,add_generation_prompt=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myfrienderic/miniconda3/envs/ktrans3.0-fix/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 1695, in apply_chat_template
rendered_chat = compiled_template.render(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myfrienderic/miniconda3/envs/ktrans3.0-fix/lib/python3.11/site-packages/jinja2/environment.py", line 1304, in render
self.environment.handle_exception()
File "/home/myfrienderic/miniconda3/envs/ktrans3.0-fix/lib/python3.11/site-packages/jinja2/environment.py", line 939, in handle_exception
raise rewrite_traceback_stack(source=source)
File "", line 3, in top-level template code
TypeError: argument of type 'NoneType' is not iterable
```
Has anyone else run into this or has any tips on how best to debug this?
Thanks!
### Reproduction
python ktransformers/server/main.py --model_path /mnt/home_extend/models/data/DeepSeek-R1-0528 --gguf_path /mnt/home_extend/models/unsloth_DeepSeek-R1-0528-GGUF/Q4_K_M --model_name DeepSeek-R1 --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml --cpu_infer 44 --max_new_tokens 30000 --cache_lens 120000 --chunk_size 512 --max_batch_size 4 --backend_type balance_serve --port 8088 --host 10.0.0.5
### Environment
Conda
Contributor guide
Research direction
Start with balance_serve.py around format_and_tokenize_input_ids at line 413 and inference at line 427, then trace the chat_completion call in chat.py at line 436. Reproduce with the provided DeepSeek-R1 command and inspect the tokenizer's apply_chat_template failure. Done means function-calling input renders without the NoneType error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100