[Bug] llama3.2 -11b-version batch infer text-only item return Garbled code

### Checklist

- [ ] 1. I have searched related issues but cannot get the expected help.
- [ ] 2. The bug has not been fixed in the latest version.
- [ ] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

### Describe the bug

As for llama3.2 -11b-version，when append a batch input, the text-only item return Garbled code.

### Reproduction

```shell
Python 3.10.12 (main, Mar 22 2024, 16:50:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from lmdeploy import pipeline, ChatTemplateConfig, PytorchEngineConfig
>>> from lmdeploy.vl import load_image
>>> pipe = pipeline('Llama-3.2-11B-Vision-Instruct')
2024-12-10 09:13:07,361 - lmdeploy - WARNING - archs.py:53 - Fallback to pytorch engine because `Llama-3.2-11B-Vision-Instruct` not supported by turbomind engine.
2024-12-10 09:13:13,244 - lmdeploy - WARNING - __init__.py:165 - LMDeploy requires transformers version: [4.33.0 ~ 4.44.1], but found version: 4.47.0
>>>
>>>
>>> image = load_image('https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg')
>>> bain = []
>>> bain.append(('what is in the image?', image))
>>> bain.append(('who won the world cup in 2006?'))
>>> response = pipe(bain)
2024-12-10 09:14:42,261 - lmdeploy - WARNING - async_engine.py:505 - GenerationConfig: GenerationConfig(n=1, max_new_tokens=512, do_sample=False, top_p=1.0, top_k=50, min_p=0.0, temperature=0.8, repetition_penalty=1.0, ignore_eos=False, random_seed=None, stop_words=None, bad_words=None, stop_token_ids=[128009, 128001, 128008], bad_token_ids=None, min_new_tokens=None, skip_special_tokens=True, logprobs=None, response_format=None, logits_processors=None)
2024-12-10 09:14:42,262 - lmdeploy - WARNING - async_engine.py:506 - Since v0.6.0, lmdeploy add `do_sample` in GenerationConfig. It defaults to False, meaning greedy decoding. Please set `do_sample=True` if sampling decoding is needed
2024-12-10 09:14:42,264 - lmdeploy - WARNING - tokenizer.py:629 - Detected duplicate bos token 128000 in prompt, this will likely reduce response quality, one of them will beremoved
>>>
>>>
>>>
>>> response
[Response(text='The image shows a tiger lying in the grass.', generate_token_len=10, input_token_len=48, session_id=0, finish_reason='stop', token_ids=[791, 2217, 5039, 264, 52835, 21078, 304, 279, 16763, 13], logprobs=None, index=0), Response(text="Italy won the 2006 Brier'aura plate. They won the FIFA Wear ' is Snitha 2006 FIFA World Cup held in Germany.", generate_token_len=32, input_token_len=51, session_id=1, finish_reason='stop', token_ids=[71603, 2834, 279, 220, 1049, 21, 426, 7401, 6, 22290, 12235, 13, 2435, 2834, 279, 44742, 42162, 364, 374, 13358, 411, 64, 220, 1049, 21, 44742, 4435, 11098, 5762, 304, 10057, 13], logprobs=None, index=1)]
>>> response[0]
Response(text='The image shows a tiger lying in the grass.', generate_token_len=10, input_token_len=48, session_id=0, finish_reason='stop', token_ids=[791, 2217, 5039, 264, 52835, 21078, 304, 279, 16763, 13], logprobs=None, index=0)
>>> response[1]
Response(text="Italy won the 2006 Brier'aura plate. They won the FIFA Wear ' is Snitha 2006 FIFA World Cup held in Germany.", generate_token_len=32, input_token_len=51, session_id=1, finish_reason='stop', token_ids=[71603, 2834, 279, 220, 1049, 21, 426, 7401, 6, 22290, 12235, 13, 2435, 2834, 279, 44742, 42162, 364, 374, 13358, 411, 64, 220, 1049, 21, 44742, 4435, 11098, 5762, 304, 10057, 13], logprobs=None, index=1)
>>> pipe(("who win the world cup in 2006?"))
Response(text='Italy won the 2006 FIFA World Cup, which was held in Germany. They defeated France 5-3 in a penalty shootout after the match ended 1-1 after extra time in the final on July 9, 2006.', generate_token_len=50, input_token_len=51, session_id=2, finish_reason='stop', token_ids=[71603, 2834, 279, 220, 1049, 21, 44742, 4435, 11098, 11, 902, 574, 5762, 304, 10057, 13, 2435, 24164, 9822, 220, 20, 12, 18, 304, 264, 16750, 88281, 1306, 279, 2489, 9670, 220, 16, 12, 16, 1306, 5066, 892, 304, 279, 1620, 389, 5887, 220, 24, 11, 220, 1049, 21, 13], logprobs=None, index=0)

```

### Environment

```Shell
pip list | grep lmde
lmdeploy 0.6.4
```

### Error traceback

_No response_