vllm + chatglm3, it can't generate content as expected
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.5k
- Forks
- 4.8k
- PR merge metrics
- No merged PRs in 30d
Description
When using vllm + ChatGLM3 and accessing the openai API, it cannot generate content. The reason is that:
1. In `conversation.py`, the ChatGLM3 template has a `stop_token_ids` with ID `2`:
```python
register_conv_template(
Conversation(
name="chatglm3",
system_template="<|system|>\n {system_message}",
roles=("<|user|>", "<|assistant|>"),
sep_style=SeparatorStyle.CHATGLM3,
stop_token_ids=[
64795,
64797,
2,
], # "<|user|>", "<|observation|>", ""
)
)
```
2. The `generate_stream` method in `vllm_worker.py` converts `stop_token_ids` into stop words. The token with ID `2` becomes an empty string after decoding:
```python
for tid in stop_token_ids:
if tid is not None:
stop.add(self.tokenizer.decode(tid))
```
Then the `SamplingParams` are:
```python
SamplingParams(n=1, best_of=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.5, top_p=1.0, top_k=-1, min_p=0.0, use_beam_search=False, length_penalty=1.0, early_stopping=False, stop=['', '<|observation|>', '<|user|>'], stop_token_ids=[64795, 64797, 2, 2], ignore_eos=False, max_tokens=32750, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True)
```
The param `stop` contains '', so it cannot generate normally.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with conversation.py to inspect the ChatGLM3 stop_token_ids and vllm_worker.py to trace how those IDs become stop words. Reproduce the issue through the OpenAI API and verify that SamplingParams no longer contains an empty stop string while generation completes normally.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai-infra-agents, api, backend
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100