[Bug] use openai server, request get asyncio.exceptions.TimeoutError

### Checklist

- [X] 1. I have searched related issues but cannot get the expected help.
- [X] 2. The bug has not been fixed in the latest version.
- [x] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

### Describe the bug

when i use openai server to start gemma-2-2b model .
```python
chat_template_config = ChatTemplateConfig('gemma')
backend_config = PytorchEngineConfig(tp=1,
session_len=3072,
max_batch_size=48,
cache_max_entry_count=0.8,
enable_prefix_caching=True,
max_prefill_token_num=2048)
serve(model_path=model_path,
model_name='gemma2_2b_it_prod',
backend='pytorch',
log_level='DEBUG',
backend_config=backend_config,
chat_template_config=chat_template_config,
server_port=8000)
```

http curl /v1/chat/completions. i get error log
```
2024-08-22 16:22:29,397 - lmdeploy - DEBUG - Engine loop is not alive.
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/opt/conda/lib/python3.8/site-packages/lmdeploy/pytorch/engine/request.py", line 171, in __no_threadsafe_get
return await asyncio.wait_for(self.resp_que.get(), timeout)
File "/opt/conda/lib/python3.8/asyncio/tasks.py", line 501, in wait_for
raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/conda/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 406, in run_asgi
result = await app( # type: ignore[func-returns-value]
File "/opt/conda/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
return await self.app(scope, receive, send)
File "/opt/conda/lib/python3.8/site-packages/fastapi/applications.py", line 1054, in __call__
await super().__call__(scope, receive, send)
File "/opt/conda/lib/python3.8/site-packages/starlette/applications.py", line 123, in __call__
await self.middleware_stack(scope, receive, send)
File "/opt/conda/lib/python3.8/site-packages/starlette/middleware/errors.py", line 164, in __call__
await self.app(scope, receive, _send)
File "/opt/conda/lib/python3.8/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/opt/conda/lib/python3.8/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/opt/conda/lib/python3.8/site-packages/starlette/routing.py", line 754, in __call__
await self.middleware_stack(scope, receive, send)
INFO: 10.104.137.121:42538 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
File "/opt/conda/lib/python3.8/site-packages/starlette/routing.py", line 774, in app
await route.handle(scope, receive, send)
File "/opt/conda/lib/python3.8/site-packages/starlette/routing.py", line 295, in handle
await self.app(scope, receive, send)
File "/opt/conda/lib/python3.8/site-packages/starlette/routing.py", line 77, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/opt/conda/lib/python3.8/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/opt/conda/lib/python3.8/site-packages/starlette/routing.py", line 74, in app
response = await f(request)
File "/opt/conda/lib/python3.8/site-packages/fastapi/routing.py", line 278, in app
raw_response = await run_endpoint_function(
File "/opt/conda/lib/python3.8/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
return await dependant.call(**values)
File "/root/app/lmdeploy_main.py", line 365, in chat_completions_v1
async for res in result_generator:
File "/opt/conda/lib/python3.8/site-packages/lmdeploy/serve/async_engine.py", line 620, in generate
async for outputs in generator.async_stream_infer(
File "/opt/conda/lib/python3.8/site-packages/lmdeploy/pytorch/engine/engine_instance.py", line 177, in async_stream_infer
resp = await self.req_sender.async_recv(req_id)
File "/opt/conda/lib/python3.8/site-packages/lmdeploy/pytorch/engine/request.py", line 314, in async_recv
resp: Response = await self._async_resp_get()
File "/opt/conda/lib/python3.8/site-packages/lmdeploy/pytorch/engine/request.py", line 187, in _async_resp_get
return await __no_threadsafe_get()
File "/opt/conda/lib/python3.8/site-packages/lmdeploy/pytorch/engine/request.py", line 175, in __no_threadsafe_get
exit(1)
File "/opt/conda/lib/python3.8/_sitebuiltins.py", line 26, in __call__
raise SystemExit(code)
SystemExit: 1
```

lmdeploy.pytorch.engine.request.RequestSender._async_resp_get
```python
async def _async_resp_get(self):
"""get resp.

Different behavior in threadsafe mode.
"""
timeout = 1

async def __no_threadsafe_get():
while True:
try:
return await asyncio.wait_for(self.resp_que.get(), timeout)
except asyncio.TimeoutError:
if not self.manager.is_loop_alive():
logger.debug('Engine loop is not alive.')
exit(1)
continue
except Exception as e:
logger.exception(
f'sender[{self.sender_id}] get response failed: {e}')
raise e

if self.is_thread_safe():
ret = self._resp_get()
await asyncio.sleep(0)
return ret
else:
return await __no_threadsafe_get()
```
why self.manager.is_loop_alive() is not alive?

### Reproduction

start command like
lmdeploy serve api_server xxxxxx

### Environment

```Shell
lmdeploy 0.5.3
```

### Error traceback

_No response_