[Bug] cpu 100%

### Checklist

- [ ] 1. I have searched related issues but cannot get the expected help.
- [ ] 2. The bug has not been fixed in the latest version.
- [ ] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

### Describe the bug


### Reproduction

lmdeploy serve api_server InternVL2-2B --backend turbomind --server-port 23333 --cache-max-entry-count 0.1 --quant-policy 4

### Environment

```Shell
4080
```

### Error traceback

_No response_