poor performance of httpx.AsyncClient in openai_api_server.py
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.5k
- Forks
- 4.8k
- PR merge metrics
- No merged PRs in 30d
Description
Hi,
While I stress "/v1/completions" interface provided by fastchat openai api server, I find its concurrency performance is very poor.
I add some debug message in process function [create_completion](https://github.com/lm-sys/FastChat/blob/main/fastchat/serve/openai_api_server.py#L458), and find when 100 request is sent in the mean time, the request would be processed nearly sequential in this function, which made response very slow.
I try change httpx.AsyncClient post method into aiohttp, and get the performance boost in this way.
So for the performance, any idea for why httpx.AsyncClient works poor in the high concurrency mode?
If needed, I could push my PR to switch httpx.AsyncClient into aiohttp.
Thx
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in fastchat/serve/openai_api_server.py at create_completion, then reproduce the reported load against the /v1/completions endpoint with concurrent requests. Compare the httpx.AsyncClient behavior with the reported aiohttp result and identify whether the client choice explains the near-sequential processing. Done means the cause is established and a verified performance fix or clear resolution is documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100