When I did the stress test, I got an error “attached to a different loop”

Nobody has claimed this yet.

INFO: 10.12.25.5:59490 - "POST /worker_generate_stream HTTP/1.1" 500 Internal Server Error
2024-01-29 10:34:12 | INFO | httpx | HTTP Request: POST http://10.12.25.5:21009/worker_generate_stream "HTTP/1.1 500 Internal Server Error"
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "E:\tools\anaconda3\envs\venv\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 408, in run_asgi
result = await app( # type: ignore[func-returns-value]
File "E:\tools\anaconda3\envs\venv\lib\site-packages\uvicorn\middleware\proxy_headers.py", line 84, in __call__
return await self.app(scope, receive, send)
File "E:\tools\anaconda3\envs\venv\lib\site-packages\fastapi\applications.py", line 1054, in __call__
await super().__call__(scope, receive, send)
File "E:\tools\anaconda3\envs\venv\lib\site-packages\starlette\applications.py", line 116, in __call__
await self.middleware_stack(scope, receive, send)
File "E:\tools\anaconda3\envs\venv\lib\site-packages\starlette\middleware\errors.py", line 186, in __call__
raise exc
File "E:\tools\anaconda3\envs\venv\lib\site-packages\starlette\middleware\errors.py", line 164, in __call__
await self.app(scope, receive, _send)
File "E:\tools\anaconda3\envs\venv\lib\site-packages\starlette\middleware\exceptions.py", line 62, in __call__
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "E:\tools\anaconda3\envs\venv\lib\site-packages\starlette\_exception_handler.py", line 55, in wrapped_app
raise exc
File "E:\tools\anaconda3\envs\venv\lib\site-packages\starlette\_exception_handler.py", line 44, in wrapped_app
await app(scope, receive, sender)
File "E:\tools\anaconda3\envs\venv\lib\site-packages\starlette\routing.py", line 746, in __call__
await route.handle(scope, receive, send)
File "E:\tools\anaconda3\envs\venv\lib\site-packages\starlette\routing.py", line 288, in handle
await self.app(scope, receive, send)
File "E:\tools\anaconda3\envs\venv\lib\site-packages\starlette\routing.py", line 75, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "E:\tools\anaconda3\envs\venv\lib\site-packages\starlette\_exception_handler.py", line 55, in wrapped_app
raise exc
File "E:\tools\anaconda3\envs\venv\lib\site-packages\starlette\_exception_handler.py", line 44, in wrapped_app
await app(scope, receive, sender)
File "E:\tools\anaconda3\envs\venv\lib\site-packages\starlette\routing.py", line 70, in app
response = await func(request)
File "E:\tools\anaconda3\envs\venv\lib\site-packages\fastapi\routing.py", line 299, in app
raise e
File "E:\tools\anaconda3\envs\venv\lib\site-packages\fastapi\routing.py", line 294, in app
raw_response = await run_endpoint_function(
File "E:\tools\anaconda3\envs\venv\lib\site-packages\fastapi\routing.py", line 191, in run_endpoint_function
return await dependant.call(**values)
File "E:\tools\anaconda3\envs\venv\lib\site-packages\fastchat\serve\base_model_worker.py", line 196, in api_generate_stream
await acquire_worker_semaphore()
File "E:\tools\anaconda3\envs\venv\lib\asyncio\locks.py", line 496, in acquire
await fut
RuntimeError: Task cb=[set.discard()]> got Future attached to a different loop

Start in fastchat/serve/base_model_worker.py at api_generate_stream and the acquire_worker_semaphore call shown in the traceback. Reproduce the stress-test request pattern and inspect the semaphore behavior across concurrent requests. Done means the endpoint handles the reported load without a different-loop RuntimeError or 500 response.

Written by the indexing model from the issue text.