lm-sys / lm-sys/FastChat

convert sep_style to int raise ValueError in some case during base_model_worker init

Open
#2,991 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
39.5k
Forks
4.8k
PR merge metrics
No merged PRs in 30d

Description

* Environment
* Python 3.11.5
* FastChat 0.2.35
* llm model: gpt-3.5-turbo
* Error logs
```
2024-01-31 20:35:54 | ERROR | stderr | Process model_worker - gpt-3.5-turbo:
2024-01-31 20:35:54 | ERROR | stderr | Traceback (most recent call last):
2024-01-31 20:35:54 | ERROR | stderr | File "/root/miniconda3/envs/codefuse/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
2024-01-31 20:35:54 | ERROR | stderr | self.run()
2024-01-31 20:35:54 | ERROR | stderr | File "/root/miniconda3/envs/codefuse/lib/python3.11/multiprocessing/process.py", line 108, in run
2024-01-31 20:35:54 | ERROR | stderr | self._target(*self._args, **self._kwargs)
2024-01-31 20:35:54 | ERROR | stderr | File "/root/codefuse-chatbot/examples/llm_api.py", line 564, in run_model_worker
2024-01-31 20:35:54 | ERROR | stderr | app = create_model_worker_app(log_level=log_level, **kwargs)
2024-01-31 20:35:54 | ERROR | stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-01-31 20:35:54 | ERROR | stderr | File "/root/codefuse-chatbot/examples/llm_api.py", line 392, in create_model_worker_app
2024-01-31 20:35:54 | ERROR | stderr | worker = ModelWorker(
2024-01-31 20:35:54 | ERROR | stderr | ^^^^^^^^^^^^
2024-01-31 20:35:54 | ERROR | stderr | File "/root/miniconda3/envs/codefuse/lib/python3.11/site-packages/fastchat/serve/model_worker.py", line 66, in __init__
2024-01-31 20:35:54 | ERROR | stderr | super().__init__(
2024-01-31 20:35:54 | ERROR | stderr | File "/root/miniconda3/envs/codefuse/lib/python3.11/site-packages/fastchat/serve/base_model_worker.py", line 48, in __init__
2024-01-31 20:35:54 | ERROR | stderr | self.conv.sep_style = int(self.conv.sep_style)
2024-01-31 20:35:54 | ERROR | stderr | ^^^^^^^^^^^^^^^^^^^^^^^^
```
* When i debug the code, i found that in `conversation.py`, function register_conv_template for ChatGPT (as same as Perplexity ai and Bard) sets the sep_style to None by default, which is the root cause the Error above
```
# ChatGPT default template
register_conv_template(
Conversation(
name="chatgpt",
system_message="You are a helpful assistant.",
roles=("user", "assistant"),
sep_style=None,
sep=None,
)
)

# Perplexity AI template
register_conv_template(
Conversation(
name="pplxai",
system_message="Be precise and concise.",
roles=("user", "assistant"),
sep_style=None,
sep=None,
)
)
```

Is it a bug needs to be fixed?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with conversation.py and the register_conv_template definitions for ChatGPT and Perplexity AI, then trace the initialization in base_model_worker.py where sep_style is converted to an integer. Check how the model worker in examples/llm_api.py supplies these templates. Done means initialization no longer raises ValueError for templates whose sep_style is None, with the intended template behavior preserved.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.