ByteDance-Seed / ByteDance-Seed/seed-oss

Throw exception when request with tool calls in vllm

Open
#10 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
891
Forks
53
PR merge metrics
No merged PRs in 30d

Description

It seem there are something wrong in the chat_template.jinja.

^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] Error in preprocessing prompt inputs^M^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] Traceback (most recent call last):^M^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] File "/data/miniconda3/envs/env-3.10/lib/python3.10/site-packages/vllm/entrypoints/chat_utils.py", line 1290, in apply_hf_chat_template^M
^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] return tokenizer.apply_chat_template(^M^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] File "/data/miniconda3/envs/env-3.10/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1640, in apply_chat_template^M^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] rendered_chat, generation_indices = render_jinja_template(^M
^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] File "/data/miniconda3/envs/env-3.10/lib/python3.10/site-packages/transformers/utils/chat_template_utils.py", line 521, in render_jinja_template^M
^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] rendered_chat = compiled_template.render(^M
^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] File "/data/miniconda3/envs/env-3.10/lib/python3.10/site-packages/jinja2/environment.py", line 1295, in render^M
^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] self.environment.handle_exception()^M^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] File "/data/miniconda3/envs/env-3.10/lib/python3.10/site-packages/jinja2/environment.py", line 942, in handle_exception^M^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] raise rewrite_traceback_stack(source=source)^M
^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] File "", line 72, in top-level template code^M^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] File "/data/miniconda3/envs/env-3.10/lib/python3.10/site-packages/jinja2/sandbox.py", line 319, in getattr^M
^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] value = getattr(obj, attribute)^M ^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'properties'^M
^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] ^M^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] The above exception was the direct cause of the following exception:^M
^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] ^M
^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] Traceback (most recent call last):^M^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] File "/data/miniconda3/envs/env-3.10/lib/python3.10/site-packages/vllm/entrypoints/openai/serving_chat.py", line 225, in create_chat_completion^M^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] ) = await self._preprocess_chat(^M
^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] File "/data/miniconda3/envs/env-3.10/lib/python3.10/site-packages/vllm/entrypoints/openai/serving_engine.py", line 896, in _preprocess_chat^M
^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] request_prompt = apply_hf_chat_template(^M
^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] File "/data/miniconda3/envs/env-3.10/lib/python3.10/site-packages/vllm/entrypoints/chat_utils.py", line 1306, in apply_hf_chat_template^M
^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] raise ValueError(str(e)) from e^M^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] ValueError: 'dict object' has no attribute 'properties'

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with chat_template.jinja at template line 72 and trace the tool-call request through serving_chat.py, serving_engine.py, and chat_utils.py. Reproduce the request that raises the missing `properties` error, then verify that tool-call chat completion no longer fails during template preprocessing.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.