Why did the model not understand the previous conversations?

### Checklist

- [X] 1. I have searched related issues but cannot get the expected help.
- [X] 2. The bug has not been fixed in the latest version.

### Describe the bug

Why did the model not understand the previous conversations? I gave OpenGVLab/InternVL-Chat-V1-5 some previous conversations with some wrong answers and when I asked "Do you think that in all the previous conversations we had, your answers were correct?" It generated the wrong response and said that No previous conversations were given.

What's the problem?

### Reproduction

Message:
`[{'role': 'system', 'content': 'Your name is Alan'}, {'role': 'user', 'content': [{'type': 'text', 'text': 'What is this place?'}, {'type': 'image_url', 'image_url': {'url': 'https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg'}}]}, {'role': 'assistant', 'content': [{'type': 'text', 'text': "It's London"}]}, {'role': 'user', 'content': [{'type': 'text', 'text': 'What is this place?'}, {'type': 'image_url', 'image_url': {'url': 'https://cdn.britannica.com/68/170868-050-8DDE8263/Golden-Gate-Bridge-San-Francisco.jpg'}}]}, {'role': 'assistant', 'content': [{'type': 'text', 'text': "It's Delhi"}]}, {'role': 'user', 'content': [{'type': 'text', 'text': 'Do you think that in all the previous conversations we had, your answers were correct? If not, where were these images taken?'}, {'type': 'image_url', 'image_url': {'url': 'https://cdn.britannica.com/59/94459-050-DBA42467/Skyline-Chicago.jpg'}}]}]`

Code:
```
class Model:
@modal.enter()
def start_engine(self):
import torch
from lmdeploy import serve, ChatTemplateConfig
print(MODEL_NAME)
self.server = serve("OpenGVLab/InternVL-Chat-V1-5",
chat_template_config=ChatTemplateConfig(
model_name='internvl-internlm2'),
server_name='0.0.0.0',
server_port=23333)

@modal.method()
async def generate(self, messages):
from lmdeploy import client
handle = client(api_server_url='http://0.0.0.0:23333')
model_name = handle.available_models[0]
print(model_name)
outputs = handle.chat_completions_v1(
model=model_name, messages=messages)
print(outputs)
for out in outputs:
return out
```

I'm building the package from GitHub