[Bug] SystemExit: 1 asyncio.exceptions.TimeoutError

### Checklist

- [X] 1. I have searched related issues but cannot get the expected help.
- [X] 2. The bug has not been fixed in the latest version.
- [ ] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

### Describe the bug

INFO: 10.12.17.5:64148 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/root/miniforge3/envs/LMDeploy/lib/python3.10/asyncio/queues.py", line 159, in get
await getter
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/root/miniforge3/envs/LMDeploy/lib/python3.10/asyncio/tasks.py", line 456, in wait_for
return fut.result()
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/root/miniforge3/envs/LMDeploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/request.py", line 171, in __no_threadsafe_get
return await asyncio.wait_for(self.resp_que.get(), timeout)
File "/root/miniforge3/envs/LMDeploy/lib/python3.10/asyncio/tasks.py", line 458, in wait_for
raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/root/miniforge3/envs/LMDeploy/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
result = await app( # type: ignore[func-returns-value]
File "/root/miniforge3/envs/LMDeploy/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
return await self.app(scope, receive, send)
File "/root/miniforge3/envs/LMDeploy/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
await super().__call__(scope, receive, send)
File "/root/miniforge3/envs/LMDeploy/lib/python3.10/site-packages/starlette/applications.py", line 123, in __call__
await self.middleware_stack(scope, receive, send)
File "/root/miniforge3/envs/LMDeploy/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in __call__
await self.app(scope, receive, _send)
File "/root/miniforge3/envs/LMDeploy/lib/python3.10/site-packages/starlette/middleware/cors.py", line 85, in __call__
await self.app(scope, receive, send)
File "/root/miniforge3/envs/LMDeploy/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/root/miniforge3/envs/LMDeploy/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/root/miniforge3/envs/LMDeploy/lib/python3.10/site-packages/starlette/routing.py", line 756, in __call__
await self.middleware_stack(scope, receive, send)
File "/root/miniforge3/envs/LMDeploy/lib/python3.10/site-packages/starlette/routing.py", line 776, in app
await route.handle(scope, receive, send)
File "/root/miniforge3/envs/LMDeploy/lib/python3.10/site-packages/starlette/routing.py", line 297, in handle
await self.app(scope, receive, send)
File "/root/miniforge3/envs/LMDeploy/lib/python3.10/site-packages/starlette/routing.py", line 77, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/root/miniforge3/envs/LMDeploy/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/root/miniforge3/envs/LMDeploy/lib/python3.10/site-packages/starlette/routing.py", line 72, in app
response = await func(request)
File "/root/miniforge3/envs/LMDeploy/lib/python3.10/site-packages/fastapi/routing.py", line 278, in app
raw_response = await run_endpoint_function(
File "/root/miniforge3/envs/LMDeploy/lib/python3.10/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
return await dependant.call(**values)
File "/root/miniforge3/envs/LMDeploy/lib/python3.10/site-packages/lmdeploy/serve/openai/api_server.py", line 483, in chat_completions_v1
async for res in result_generator:
File "/root/miniforge3/envs/LMDeploy/lib/python3.10/site-packages/lmdeploy/serve/async_engine.py", line 575, in generate
async for outputs in generator.async_stream_infer(
File "/root/miniforge3/envs/LMDeploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/engine_instance.py", line 178, in async_stream_infer
resp = await self.req_sender.async_recv(req_id)
File "/root/miniforge3/envs/LMDeploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/request.py", line 312, in async_recv
resp: Response = await self._async_resp_get()
File "/root/miniforge3/envs/LMDeploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/request.py", line 187, in _async_resp_get
return await __no_threadsafe_get()
File "/root/miniforge3/envs/LMDeploy/lib/python3.10/site-packages/lmdeploy/pytorch/engine/request.py", line 175, in __no_threadsafe_get
exit(1)
File "/root/miniforge3/envs/LMDeploy/lib/python3.10/_sitebuiltins.py", line 26, in __call__
raise SystemExit(code)
SystemExit: 1

### Reproduction

CUDA_VISIBLE_DEVICES=6 lmdeploy serve api_server /workspace/model/llm/Qwen/Qwen2_5-3B-Instruct \
--server-port 2016 \
--tp 1 \
--cache-max-entry-count 0.2 \
--session-len 10000 \
--enable-prefix-caching \
--adapters rewrite_3B=/workspace/output/question_extration/qwen/qwen2_5-3b-instruct/v0-20241119-170654/checkpoint-1200 \
--enable-prefix-caching \
--chat-template qwen \
--backend pytorch \

### Environment

```Shell
lmdeploy 0.6.3
```

### Error traceback

_No response_