vllm api will randomly generate answers slowly when applying asynchronous parallel calls
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.5k
- Forks
- 4.8k
- PR merge metrics
- No merged PRs in 30d
Description
fschat = 0.2.36 0.2.34
I serve the api using the following steps:
```
python -m fastchat.serve.controller
python -m fastchat.serve.vllm_worker
python -m fastchat.serve.openai_api_server
```
I call the api asynchronously for 15 prompts. For the same 15 prompts, sometimes the response time is around 4s, while sometimes it will be 40-50s. And the model answers are the same for the two different situations.
What causes this problem ?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the issue with the three Python commands in the report and 15 asynchronous prompts, comparing the reported 4-second and 40–50-second response times. The payload names no source files or tests, so trace the serving path across the controller, vLLM worker, and OpenAI API server. Done means identifying the cause of the variable latency and documenting a reproducible diagnosis.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, api, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100