[Bug] internvl 模型被推理后，针对图片内容回答的答案不正确

### Checklist

- [x] 1. I have searched related issues but cannot get the expected help.
- [ ] 2. The bug has not been fixed in the latest version.

### Describe the bug

相同模型InternVL-Chat-V1-5 ,未加速时回复正确，但是通过加速推理后，针对图片的回答不稳定/不正确

### Reproduction

from lmdeploy import pipeline, GenerationConfig, TurbomindEngineConfig
from lmdeploy.vl import load_image

pipe = pipeline('models/InternVL-Chat-V1-5',
backend_config=TurbomindEngineConfig(tp=1))
gen_config = GenerationConfig(temperature=0.6, max_new_tokens=512)
image = load_image('https://xxxxxxxx.jpeg')
response = pipe(('三类职业的每人年保费是多少？', image), gen_config=gen_config)
print(response)

### Environment

```Shell
lmdeploy==0.4.2
```

### Error traceback

_No response_