[Bug] crash when sleep but not wakeup in version 0.11.0

### Checklist

- [x] 1. I have searched related issues but cannot get the expected help.
- [x] 2. The bug has not been fixed in the latest version.
- [x] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

### Describe the bug

i start a server and try sleep wakeup api, see crash

### Reproduction

step1
[run_server.py](https://github.com/user-attachments/files/23931475/run_server.py)
step2
[test_time.py](https://github.com/user-attachments/files/23931477/test_time.py)
then wait crash

### Environment

```Shell
lmdeploy==0.11.0
```

### Error traceback

```Shell
HINT: Please open http://127.0.0.1:23333 in a browser for detailed api usage!!!
HINT: Please open http://127.0.0.1:23333 in a browser for detailed api usage!!!
HINT: Please open http://127.0.0.1:23333 in a browser for detailed api usage!!!
INFO: Started server process [226438]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://127.0.0.1:23333 (Press CTRL+C to quit)
INFO: 127.0.0.1:60974 - "POST /sleep?level=1 HTTP/1.1" 200 OK
INFO: 127.0.0.1:32806 - "POST /wakeup HTTP/1.1" 200 OK
INFO: 127.0.0.1:33350 - "POST /sleep?level=1 HTTP/1.1" 200 OK
INFO: 127.0.0.1:33362 - "POST /wakeup HTTP/1.1" 200 OK
INFO: 127.0.0.1:33970 - "POST /sleep?level=1 HTTP/1.1" 200 OK
Segmentation fault
```