vllm_worker is incompatible with vllm > 0.8.x
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.5k
- Forks
- 4.8k
- PR merge metrics
- No merged PRs in 30d
Description
While this works on VLLM 0.7.x, the latest one (0.8.2), which supports mistral-small and gemma, does not seem have the "engine" attribute:
```python
2025-03-24 22:04:27 | ERROR | stderr | Traceback (most recent call last):
2025-03-24 22:04:27 | ERROR | stderr | File "/p/haicluster/llama/FastChat/fastchat/serve/vllm_worker.py", line 291, in
2025-03-24 22:04:27 | ERROR | stderr | worker = VLLMWorker(
2025-03-24 22:04:27 | ERROR | stderr | ^^^^^^^^^^^
2025-03-24 22:04:27 | ERROR | stderr | File "/p/haicluster/llama/FastChat/fastchat/serve/vllm_worker.py", line 57, in __init__
2025-03-24 22:04:27 | ERROR | stderr | self.tokenizer = llm_engine.engine.tokenizer
2025-03-24 22:04:27 | ERROR | stderr | ^^^^^^^^^^^^^^^^^
2025-03-24 22:04:27 | ERROR | stderr | AttributeError: 'AsyncLLM' object has no attribute 'engine'
```
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in fastchat/serve/vllm_worker.py around line 57, where initialization accesses llm_engine.engine.tokenizer, and compare the VLLM 0.7.x and 0.8.2 behavior described in the traceback. Confirm the worker can initialize with VLLM 0.8.2 without the missing-attribute error while preserving the tokenizer setup.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100