ByteDance-Seed / ByteDance-Seed/seed-oss
Throw exception when request with tool calls in vllm
- Dominant language
- Python
- Stars
- 891
- Forks
- 53
- PR merge metrics
- No merged PRs in 30d
Description
It seem there are something wrong in the chat_template.jinja.
^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] Error in preprocessing prompt inputs^M^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] Traceback (most recent call last):^M^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] File "/data/miniconda3/envs/env-3.10/lib/python3.10/site-packages/vllm/entrypoints/chat_utils.py", line 1290, in apply_hf_chat_template^M
^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] return tokenizer.apply_chat_template(^M^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] File "/data/miniconda3/envs/env-3.10/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1640, in apply_chat_template^M^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] rendered_chat, generation_indices = render_jinja_template(^M
^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] File "/data/miniconda3/envs/env-3.10/lib/python3.10/site-packages/transformers/utils/chat_template_utils.py", line 521, in render_jinja_template^M
^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] rendered_chat = compiled_template.render(^M
^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] File "/data/miniconda3/envs/env-3.10/lib/python3.10/site-packages/jinja2/environment.py", line 1295, in render^M
^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] self.environment.handle_exception()^M^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] File "/data/miniconda3/envs/env-3.10/lib/python3.10/site-packages/jinja2/environment.py", line 942, in handle_exception^M^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] raise rewrite_traceback_stack(source=source)^M
^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] File "", line 72, in top-level template code^M^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] File "/data/miniconda3/envs/env-3.10/lib/python3.10/site-packages/jinja2/sandbox.py", line 319, in getattr^M
^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] value = getattr(obj, attribute)^M ^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'properties'^M
^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] ^M^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] The above exception was the direct cause of the following exception:^M
^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] ^M
^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] Traceback (most recent call last):^M^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] File "/data/miniconda3/envs/env-3.10/lib/python3.10/site-packages/vllm/entrypoints/openai/serving_chat.py", line 225, in create_chat_completion^M^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] ) = await self._preprocess_chat(^M
^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] File "/data/miniconda3/envs/env-3.10/lib/python3.10/site-packages/vllm/entrypoints/openai/serving_engine.py", line 896, in _preprocess_chat^M
^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] request_prompt = apply_hf_chat_template(^M
^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] File "/data/miniconda3/envs/env-3.10/lib/python3.10/site-packages/vllm/entrypoints/chat_utils.py", line 1306, in apply_hf_chat_template^M
^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] raise ValueError(str(e)) from e^M^[[1;36m(APIServer pid=679631)^[[0;0m ERROR 08-22 21:36:14 [serving_chat.py:250] ValueError: 'dict object' has no attribute 'properties'
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with chat_template.jinja at template line 72 and trace the tool-call request through serving_chat.py, serving_engine.py, and chat_utils.py. Reproduce the request that raises the missing `properties` error, then verify that tool-call chat completion no longer fails during template preprocessing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100