lmdeploy pipe 部署internvl2-40B batch_infer出错[Bug]

### Checklist

- [X] 1. I have searched related issues but cannot get the expected help.
- [X] 2. The bug has not been fixed in the latest version.
- [X] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

### Describe the bug

batch_infer出错，但是只推理一个没问题

main()
File "/checkpoint/binary/train_package/./test2.py", line 47, in main
batch_out = batch_infer(batch_input)
File "/checkpoint/binary/train_package/./test2.py", line 23, in batch_infer
batch_out = pipe(batch_input)
File "/root/.local/lib/python3.10/site-packages/lmdeploy/serve/vl_async_engine.py", line 123, in __call__
return super().__call__(prompts, **kwargs)
File "/root/.local/lib/python3.10/site-packages/lmdeploy/serve/async_engine.py", line 305, in __call__
return self.batch_infer(prompts,
File "/root/.local/lib/python3.10/site-packages/lmdeploy/serve/vl_async_engine.py", line 109, in batch_infer
return super().batch_infer(prompts, **kwargs)
File "/root/.local/lib/python3.10/site-packages/lmdeploy/serve/async_engine.py", line 429, in batch_infer
_get_event_loop().run_until_complete(gather())
File "/opt/conda/envs/python3.10.13/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
return future.result()
File "/root/.local/lib/python3.10/site-packages/lmdeploy/serve/async_engine.py", line 426, in gather
await asyncio.gather(
File "/root/.local/lib/python3.10/site-packages/lmdeploy/serve/async_engine.py", line 411, in _inner_call
async for out in generator:
File "/root/.local/lib/python3.10/site-packages/lmdeploy/serve/async_engine.py", line 635, in generate
response, state = self.tokenizer.detokenize_incrementally(
File "/root/.local/lib/python3.10/site-packages/lmdeploy/tokenizer.py", line 642, in detokenize_incrementally
return self.model.detokenize_incrementally(
File "/root/.local/lib/python3.10/site-packages/lmdeploy/tokenizer.py", line 460, in detokenize_incrementally
new_tokens = tokenizer.convert_ids_to_tokens(
File "/root/.local/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 412, in convert_ids_to_tokens
tokens.append(self._tokenizer.id_to_token(index))
OverflowError: out of range integral type conversion attempted

### Reproduction

rt

### Environment

```Shell
lmd==0.5.3
A100*2
```

### Error traceback

```Shell
main()
File "/checkpoint/binary/train_package/./test2.py", line 47, in main
batch_out = batch_infer(batch_input)
File "/checkpoint/binary/train_package/./test2.py", line 23, in batch_infer
batch_out = pipe(batch_input)
File "/root/.local/lib/python3.10/site-packages/lmdeploy/serve/vl_async_engine.py", line 123, in __call__
return super().__call__(prompts, **kwargs)
File "/root/.local/lib/python3.10/site-packages/lmdeploy/serve/async_engine.py", line 305, in __call__
return self.batch_infer(prompts,
File "/root/.local/lib/python3.10/site-packages/lmdeploy/serve/vl_async_engine.py", line 109, in batch_infer
return super().batch_infer(prompts, **kwargs)
File "/root/.local/lib/python3.10/site-packages/lmdeploy/serve/async_engine.py", line 429, in batch_infer
_get_event_loop().run_until_complete(gather())
File "/opt/conda/envs/python3.10.13/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
return future.result()
File "/root/.local/lib/python3.10/site-packages/lmdeploy/serve/async_engine.py", line 426, in gather
await asyncio.gather(
File "/root/.local/lib/python3.10/site-packages/lmdeploy/serve/async_engine.py", line 411, in _inner_call
async for out in generator:
File "/root/.local/lib/python3.10/site-packages/lmdeploy/serve/async_engine.py", line 635, in generate
response, state = self.tokenizer.detokenize_incrementally(
File "/root/.local/lib/python3.10/site-packages/lmdeploy/tokenizer.py", line 642, in detokenize_incrementally
return self.model.detokenize_incrementally(
File "/root/.local/lib/python3.10/site-packages/lmdeploy/tokenizer.py", line 460, in detokenize_incrementally
new_tokens = tokenizer.convert_ids_to_tokens(
File "/root/.local/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 412, in convert_ids_to_tokens
tokens.append(self._tokenizer.id_to_token(index))
OverflowError: out of range integral type conversion attempted
```