[Bug] `detokenize_incrementally`: OverflowError: out of range integral type conversion attempted

### Checklist

- [X] 1. I have searched related issues but cannot get the expected help.
- [X] 2. The bug has not been fixed in the latest version.

### Describe the bug

Most API requests are successful, but the error in the title randomly occurs sometimes. I haven't worked out how to reliably reproduce it yet. I'm not doing anything special - just loading a Llama 2 70B AWQ model on that I created with `lmdeploy lite auto_awq` on a dual 4090 machine.

The same model works fine in vLLM.

### Reproduction

Seems to occur randomly when many concurrent requests are sent. I will update this issue if I find any way to reproduce it consistently. Here are the arguments:
```
lmdeploy serve api_server MODEL_NAME --tp 2 --session-len 4096 --model-format awq --quant-policy 4 --model-name MODEL_NAME --enable-prefix-caching
```
I'm using the `/v1/completions` endpoint (not the chat completion endpoint). The tokenizer class is `LlamaTokenizer`.

I'm hoping that the error log below sparks an idea at what the cause could be :crossed_fingers:

### Environment

Latest official Docker image: `openmmlab/lmdeploy:v0.4.2`

### Error traceback

```Shell
2024-06-07T23:16:22.614538430Z ERROR: Exception in ASGI application
2024-06-07T23:16:22.614568648Z Traceback (most recent call last):
2024-06-07T23:16:22.614572556Z File "/opt/py38/lib/python3.8/site-packages/starlette/responses.py", line 265, in __call__
2024-06-07T23:16:22.614576223Z await wrap(partial(self.listen_for_disconnect, receive))
2024-06-07T23:16:22.614579830Z File "/opt/py38/lib/python3.8/site-packages/starlette/responses.py", line 261, in wrap
2024-06-07T23:16:22.614584258Z await func()
2024-06-07T23:16:22.614587845Z File "/opt/py38/lib/python3.8/site-packages/starlette/responses.py", line 238, in listen_for_disconnect
2024-06-07T23:16:22.614591793Z message = await receive()
2024-06-07T23:16:22.614595310Z File "/opt/py38/lib/python3.8/site-packages/uvicorn/protocols/http/httptools_impl.py", line 568, in receive
2024-06-07T23:16:22.614598726Z await self.message_event.wait()
2024-06-07T23:16:22.614602193Z File "/usr/lib/python3.8/asyncio/locks.py", line 309, in wait
2024-06-07T23:16:22.614605649Z await fut
2024-06-07T23:16:22.614609046Z asyncio.exceptions.CancelledError
2024-06-07T23:16:22.614612433Z
2024-06-07T23:16:22.614615789Z During handling of the above exception, another exception occurred:
2024-06-07T23:16:22.614619306Z
2024-06-07T23:16:22.614622622Z + Exception Group Traceback (most recent call last):
2024-06-07T23:16:22.614626079Z | File "/opt/py38/lib/python3.8/site-packages/uvicorn/protocols/http/httptools_impl.py", line 411, in run_asgi
2024-06-07T23:16:22.614629485Z | result = await app( # type: ignore[func-returns-value]
2024-06-07T23:16:22.614632882Z | File "/opt/py38/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in __call__
2024-06-07T23:16:22.614636258Z | return await self.app(scope, receive, send)
2024-06-07T23:16:22.614639645Z | File "/opt/py38/lib/python3.8/site-packages/fastapi/applications.py", line 1054, in __call__
2024-06-07T23:16:22.614643032Z | await super().__call__(scope, receive, send)
2024-06-07T23:16:22.614646458Z | File "/opt/py38/lib/python3.8/site-packages/starlette/applications.py", line 123, in __call__
2024-06-07T23:16:22.614649825Z | await self.middleware_stack(scope, receive, send)
2024-06-07T23:16:22.614653191Z | File "/opt/py38/lib/python3.8/site-packages/starlette/middleware/errors.py", line 186, in __call__
2024-06-07T23:16:22.614656588Z | raise exc
2024-06-07T23:16:22.614660014Z | File "/opt/py38/lib/python3.8/site-packages/starlette/middleware/errors.py", line 164, in __call__
2024-06-07T23:16:22.614663391Z | await self.app(scope, receive, _send)
2024-06-07T23:16:22.614666737Z | File "/opt/py38/lib/python3.8/site-packages/starlette/middleware/cors.py", line 93, in __call__
2024-06-07T23:16:22.614670124Z | await self.simple_response(scope, receive, send, request_headers=headers)
2024-06-07T23:16:22.614674382Z | File "/opt/py38/lib/python3.8/site-packages/starlette/middleware/cors.py", line 148, in simple_response
2024-06-07T23:16:22.614677849Z | await self.app(scope, receive, send)
2024-06-07T23:16:22.614681285Z | File "/opt/py38/lib/python3.8/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
2024-06-07T23:16:22.614684702Z | await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
2024-06-07T23:16:22.614688088Z | File "/opt/py38/lib/python3.8/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
2024-06-07T23:16:22.614702406Z | raise exc
2024-06-07T23:16:22.614706003Z | File "/opt/py38/lib/python3.8/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
2024-06-07T23:16:22.614709389Z | await app(scope, receive, sender)
2024-06-07T23:16:22.614712786Z | File "/opt/py38/lib/python3.8/site-packages/starlette/routing.py", line 756, in __call__
2024-06-07T23:16:22.614716153Z | await self.middleware_stack(scope, receive, send)
2024-06-07T23:16:22.614719489Z | File "/opt/py38/lib/python3.8/site-packages/starlette/routing.py", line 776, in app
2024-06-07T23:16:22.614722825Z | await route.handle(scope, receive, send)
2024-06-07T23:16:22.614726182Z | File "/opt/py38/lib/python3.8/site-packages/starlette/routing.py", line 297, in handle
2024-06-07T23:16:22.614729528Z | await self.app(scope, receive, send)
2024-06-07T23:16:22.614732885Z | File "/opt/py38/lib/python3.8/site-packages/starlette/routing.py", line 77, in app
2024-06-07T23:16:22.614736241Z | await wrap_app_handling_exceptions(app, request)(scope, receive, send)
2024-06-07T23:16:22.614739628Z | File "/opt/py38/lib/python3.8/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
2024-06-07T23:16:22.614743084Z | raise exc
2024-06-07T23:16:22.614746571Z | File "/opt/py38/lib/python3.8/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
2024-06-07T23:16:22.614749988Z | await app(scope, receive, sender)
2024-06-07T23:16:22.614753324Z | File "/opt/py38/lib/python3.8/site-packages/starlette/routing.py", line 75, in app
2024-06-07T23:16:22.614756671Z | await response(scope, receive, send)
2024-06-07T23:16:22.614760017Z | File "/opt/py38/lib/python3.8/site-packages/starlette/responses.py", line 265, in __call__
2024-06-07T23:16:22.614763454Z | await wrap(partial(self.listen_for_disconnect, receive))
2024-06-07T23:16:22.614766870Z | File "/opt/py38/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 680, in __aexit__
2024-06-07T23:16:22.614770227Z | raise BaseExceptionGroup(
2024-06-07T23:16:22.614773613Z | exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
2024-06-07T23:16:22.614776980Z +-+---------------- 1 ----------------
2024-06-07T23:16:22.614780346Z | Traceback (most recent call last):
2024-06-07T23:16:22.614784043Z | File "/opt/py38/lib/python3.8/site-packages/starlette/responses.py", line 261, in wrap
2024-06-07T23:16:22.614787440Z | await func()
2024-06-07T23:16:22.614790827Z | File "/opt/py38/lib/python3.8/site-packages/starlette/responses.py", line 250, in stream_response
2024-06-07T23:16:22.614794253Z | async for chunk in self.body_iterator:
2024-06-07T23:16:22.614797570Z | File "/opt/lmdeploy/lmdeploy/serve/openai/api_server.py", line 778, in completion_stream_generator
2024-06-07T23:16:22.614800916Z | async for res in generator:
2024-06-07T23:16:22.614804272Z | File "/opt/lmdeploy/lmdeploy/serve/async_engine.py", line 633, in generate
2024-06-07T23:16:22.614807659Z | response, state = self.tokenizer.detokenize_incrementally(
2024-06-07T23:16:22.614810985Z | File "/opt/lmdeploy/lmdeploy/tokenizer.py", line 569, in detokenize_incrementally
2024-06-07T23:16:22.614814502Z | return self.model.detokenize_incrementally(
2024-06-07T23:16:22.614817879Z | File "/opt/lmdeploy/lmdeploy/tokenizer.py", line 425, in detokenize_incrementally
2024-06-07T23:16:22.614821265Z | new_tokens = tokenizer.convert_ids_to_tokens(
2024-06-07T23:16:22.614824642Z | File "/opt/py38/lib/python3.8/site-packages/transformers/tokenization_utils_fast.py", line 399, in convert_ids_to_tokens
2024-06-07T23:16:22.614828058Z | tokens.append(self._tokenizer.id_to_token(index))
2024-06-07T23:16:22.614831415Z | OverflowError: out of range integral type conversion attempted
2024-06-07T23:16:22.614834771Z +------------------------------------

# Then after sending another request:

2024-06-07T23:17:17.695793520Z terminate called after throwing an instance of 'std::runtime_error'
2024-06-07T23:17:17.695805133Z what(): [TM][ERROR] CUDA runtime error: an illegal memory access was encountered /opt/lmdeploy/src/turbomind/utils/allocator.h:231
2024-06-07T23:17:17.695807778Z
2024-06-07T23:17:17.695809922Z terminate called recursively
```