lm-sys / lm-sys/FastChat

vllm + chatglm3, it can't generate content as expected

Open
#2,845 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
39.5k
Forks
4.8k
PR merge metrics
No merged PRs in 30d

Description

When using vllm + ChatGLM3 and accessing the openai API, it cannot generate content. The reason is that:

1. In `conversation.py`, the ChatGLM3 template has a `stop_token_ids` with ID `2`:
```python
register_conv_template(
    Conversation(
        name="chatglm3",
        system_template="<|system|>\n {system_message}",
        roles=("<|user|>", "<|assistant|>"),
        sep_style=SeparatorStyle.CHATGLM3,
        stop_token_ids=[
            64795,
            64797,
            2,
        ],  # "<|user|>", "<|observation|>", ""
    )
)
```

2. The `generate_stream` method in `vllm_worker.py` converts `stop_token_ids` into stop words. The token with ID `2` becomes an empty string after decoding:
```python
        for tid in stop_token_ids:
            if tid is not None:
                stop.add(self.tokenizer.decode(tid))
```
Then the `SamplingParams` are:
```python
SamplingParams(n=1, best_of=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.5, top_p=1.0, top_k=-1, min_p=0.0, use_beam_search=False, length_penalty=1.0, early_stopping=False, stop=['', '<|observation|>', '<|user|>'], stop_token_ids=[64795, 64797, 2, 2], ignore_eos=False, max_tokens=32750, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True)
```
The param `stop` contains '', so it cannot generate normally.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with conversation.py to inspect the ChatGLM3 stop_token_ids and vllm_worker.py to trace how those IDs become stop words. Reproduce the issue through the OpenAI API and verify that SamplingParams no longer contains an empty stop string while generation completes normally.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai-infra-agents, api, backend
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.