[Bug] It seems the second round of answers lost the context of the first round in baichun2-13b-chat models with pytorch backend or CodeLlama-7b-Instruct-hf with turbomind backend

### Checklist

- [ ] 1. I have searched related issues but cannot get the expected help.
- [ ] 2. The bug has not been fixed in the latest version.
- [ ] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

### Describe the bug

It seems the second round of answers lost the context of the first round in baichun2-13b-chat models with pytorch backend.

### Reproduction

lmdeploy chat /nvme/qa_test_models/baichuan-inc/Baichuan2-13B-Chat --tp 2
lmdeploy chat /nvme/qa_test_models/codellama/CodeLlama-7b-Instruct-hf

input 介绍乌鲁木齐的景点
input 美食呢？

the second round of answers need to be related 乌鲁木齐

### Environment

```Shell
sys.platform: linux
Python: 3.10.12 (main, Nov 6 2024, 20:22:13) [GCC 11.4.0]
CUDA available: True
MUSA available: False
numpy_random_seed: 2147483648
GPU 0,1,2,3,4,5,6,7: NVIDIA A100-SXM4-80GB
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 11.8, V11.8.89
GCC: x86_64-linux-gnu-gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
PyTorch: 2.4.1+cu118
PyTorch compiling details: PyTorch built with:
- GCC 9.3
- C++ Version: 201703
- Intel(R) oneAPI Math Kernel Library Version 2022.2-Product Build 20220804 for Intel(R) 64 architecture applications
- Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67)
- OpenMP 201511 (a.k.a. OpenMP 4.5)
- LAPACK is enabled (usually provided by MKL)
- NNPACK is enabled
- CPU capability usage: AVX512
- CUDA Runtime 11.8
- NVCC architecture flags: -gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_90,code=sm_90
- CuDNN 90.1
- Magma 2.6.1
- Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.8, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.4.1, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=1, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF,

TorchVision: 0.19.1+cu118
LMDeploy: 0.6.5+b140eb5
transformers: 4.47.1
gradio: 5.9.1
fastapi: 0.115.6
pydantic: 2.10.4
triton: 3.0.0
NVIDIA Topology:
GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 CPU Affinity NUMA Affinity
GPU0 X NV12 NV12 NV12 NV12 NV12 NV12 NV12 0-27,56-83 0
GPU1 NV12 X NV12 NV12 NV12 NV12 NV12 NV12 0-27,56-83 0
GPU2 NV12 NV12 X NV12 NV12 NV12 NV12 NV12 0-27,56-83 0
GPU3 NV12 NV12 NV12 X NV12 NV12 NV12 NV12 0-27,56-83 0
GPU4 NV12 NV12 NV12 NV12 X NV12 NV12 NV12 28-55,84-111 1
GPU5 NV12 NV12 NV12 NV12 NV12 X NV12 NV12 28-55,84-111 1
GPU6 NV12 NV12 NV12 NV12 NV12 NV12 X NV12 28-55,84-111 1
GPU7 NV12 NV12 NV12 NV12 NV12 NV12 NV12 X 28-55,84-111 1

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks
```

### Error traceback

```Shell
2025-01-13 13:44:26,298 - lmdeploy - WARNING - archs.py:53 - Fallback to pytorch engine because `/nvme/qa_test_models/baichuan-inc/Baichuan2-13B-Chat` not supported by turbomind engine.
2025-01-13 13:44:28,591 - lmdeploy - WARNING - transformers.py:22 - LMDeploy requires transformers version: [4.33.0 ~ 4.46.1], but found version: 4.47.1
2025-01-13 13:44:28,600 - lmdeploy - INFO - model_agent.py:613 - MASTER_ADDR=127.0.0.1, MASTER_PORT=29500
2025-01-13 13:44:31,731 - lmdeploy - INFO - model_agent.py:335 - build model.
2025-01-13 13:44:31,990 - lmdeploy - INFO - model_agent.py:338 - loading weights.
2025-01-13 13:44:31,990 - lmdeploy - INFO - model_weight_loader.py:142 - rank[0] loading weights - "pytorch_model-00002-of-00003.bin"
2025-01-13 13:44:32,322 - lmdeploy - INFO - model_weight_loader.py:142 - rank[1] loading weights - "pytorch_model-00002-of-00003.bin"
2025-01-13 13:44:33,852 - lmdeploy - INFO - model_weight_loader.py:142 - rank[0] loading weights - "pytorch_model-00001-of-00003.bin"
2025-01-13 13:44:34,156 - lmdeploy - INFO - model_weight_loader.py:142 - rank[1] loading weights - "pytorch_model-00003-of-00003.bin"
2025-01-13 13:44:36,178 - lmdeploy - INFO - model_weight_loader.py:142 - rank[1] loading weights - "pytorch_model-00001-of-00003.bin"
2025-01-13 13:44:36,237 - lmdeploy - INFO - model_weight_loader.py:142 - rank[0] loading weights - "pytorch_model-00003-of-00003.bin"
2025-01-13 13:44:39,463 - lmdeploy - INFO - cache_engine.py:36 - build CacheEngine with config:CacheConfig(max_batches=256, block_size=64, num_cpu_blocks=79, num_gpu_blocks=3881, window_size=-1, cache_max_entry_count=0.8, max_prefill_token_num=4096, enable_prefix_caching=False, quant_policy=8, device_type='cuda')

double enter to end input >>> 你好,介绍乌鲁木齐的景点

你好,介绍乌鲁木齐的景点乌鲁木齐是新疆维吾尔自治区的首府，也是一个历史悠久、文化多元的城市。这里有许多值得一游的景点，以下是一些建议：

1. 红山公园：红山公园位于乌鲁木齐市区的中心，海拔高度为910米。登上山顶，可以俯瞰整个乌鲁木齐市的美景。山上有一座红山塔，建于1906年，是乌鲁木齐市的地标之一。

2. 新疆博物馆：新疆博物馆是新疆维吾尔自治区最大的博物馆，馆内陈列着丰富的历史文物和艺术品，展示了新疆悠久的历史和文化。

3. 国际大巴扎：国际大巴扎位于乌鲁木齐市的天山区，是一个充满异域风情的商业区。这里有各种特色的商品、美食和手工艺品，游客可以在这里感受维吾尔族等新疆民族的风土人情。

4. 天山天池：天山天池位于乌鲁木齐市以南150公里的地方，是新疆著名的自然景观。天池四周环山，湖水清澈，景色秀美。游客可以乘坐索道登山，欣赏天池的美景。

5. 南山牧场：南山牧场位于乌鲁木齐市以南的南麓，是一个避暑纳凉的好地方。这里有草原、雪山、森林等多种景观，游客可以在这里骑马、徒步，感受大自然的美好。

6. 盐湖：盐湖位于乌鲁木齐市以东的吉木萨尔县，是一个美丽的盐湖景观。湖水呈现出独特的蓝色，与周围的环境形成鲜明的对比。游客可以在这里欣赏盐湖美景，也可以进行盐浴、盐疗等休闲活动。

7. 吐鲁番盆地：吐鲁番盆地位于乌鲁木齐市以东的吐鲁番市，是世界著名的炎热地区。这里有著名的葡萄沟、火焰山、交河城等景点，游客可以在这里感受大漠孤烟的壮美景观。

总之，乌鲁木齐有很多美丽的景点，游客可以在这里体验新疆丰富的历史、文化和自然景观。

double enter to end input >>> 美食呢?

美食呢?美食是一种令人愉悦的食物，通常包含各种口味和食材。美食可以包括各种菜肴、甜点、饮品等。不同地区和国家的美食各具特色，如中国的麻辣火锅、意大利的披萨、日本的寿司等。美食不仅仅是一种味觉享受，更是一种文化和社会现象。

double enter to end input >>> exit

lmdeploy chat /nvme/qa_test_models/codellama/CodeLlama-7b-Instruct-hf
chat_template_config:
ChatTemplateConfig(model_name='codellama', system=None, meta_instruction=None, eosys=None, user=None, eoh=None, assistant=None, eoa=None, tool=None, eotool=None, separator=None, capability='chat', stop_words=None)
engine_cfg:
TurbomindEngineConfig(dtype='auto', model_format=None, tp=1, session_len=16384, max_batch_size=1, cache_max_entry_count=0.8, cache_chunk_size=-1, cache_block_seq_len=64, enable_prefix_caching=False, quant_policy=0, rope_scaling_factor=0.0, use_logn_attn=False, download_dir=None, revision=None, max_prefill_token_num=8192, num_tokens_per_iter=0, max_prefill_iters=1)
[WARNING] gemm_config.in is not found; using default GEMM algo

enter !! to end the input >>>
介绍乌鲁木齐的景点
!!
[INST] <>

<>

介绍乌鲁木齐的景点 [/INST] Sure, here are some popular tourist attractions in Urumqi:

1. Kashgar Museum: This museum is located in
the ancient city of Kashgar and is dedicated to the history and culture of the region. It features exhibits on the Silk Road, Buddhism, and other important topics.
2. Xinjiang Museum: This museum is located in the capital city of Urumqi and features exhibits on the history and culture of Xinjiang, as well as its natural resources and economy.
3. Gurje-Tash: This ancient fortress is located in the north of Urumqi and is a UNESCO World Heritage Site. It was built in the 14th century and features a large square courtyard and a series of towers and battlements.
4. Kumquat Valley: This valley is located in the south of Urumqi and is known for its beautiful scenery and lush greenery. It is also home to a number of orchards and fruit farms.
5. Tianchi Park: This park is located in the heart of Urumqi and features a large lake, a number of water features, and a variety of recreational facilities. It is a popular spot for locals and tourists alike.
6. Alashan Tibetan Culture Museum: This museum is located in the northeastern part of Urumqi and features exhibits on the Tibetan culture and history of the region. It is a great place to learn about the unique traditions and customs of the Tibetan people.
7. Xinjiang Academy of Social Sciences: This academy is located in the heart of Urumqi and is home to a number of research institutions and academic departments. It is a great place to learn about the social sciences and humanities.
8. Urumqi Botanical Garden: This garden is located in the north of Urumqi and features a variety of plants and flowers from around the world. It is a great place to learn about botany and see a variety of different species.
9. Ming-Kuang Park: This park is located in the south of Urumqi and features a large lake, a number of water features, and a variety of recreational facilities. It is a popular spot for locals and tourists alike.
10. Urumqi International Airport: This airport is located in the southeast of Urumqi and is a major hub for China Southern Airlines and other airlines. It is a great place to learn about aviation and see the different types of aircraft that operate at the airport.

enter !! to end the input >>>
美食呢?
!!
[INST]
美食呢? [/INST] Sure, here are some popular Chinese dishes that you might enjoy:

1. Peking Duck: A dish originating from Beijing, Peking duck is a roasted duck dish that is served with pancakes, scallions, and a sweet bean sauce.
2. Kung Pao Chicken: A spicy Sichuan dish made with diced chicken, peanuts, vegetables, and chili peppers.
3. Xiaolongbao (Soup Dumplings): A type of steamed dumpling from Shanghai, filled with pork and a savory broth.
4. Hot Pot: A popular communal dish from Sichuan, hot pot is a simmering pot of broth filled with various ingredients such as meat, seafood, and vegetables.
5. Egg Tarts: A sweet and savory pastry from Taiwan, made with a flaky crust and a creamy egg custard filling.
6. Ma Po Tofu: A spicy Sichuan dish made with braised tofu, ground beef, and a variety of vegetables.
7. Wonton Noodles: A Cantonese dish made with thin noodles and wontons filled with shrimp or pork.
8. Dadan Noodles: A popular Shanghai dish made with thin noodles and a savory broth, often served with sliced pork and vegetables.
9. Mapo Tofu: A spicy Sichuan dish made with braised tofu, ground beef, and a variety of vegetables.
10. Sweet and Sour Pork: A Cantonese dish made with deep-fried pork, a sweet and sour sauce, and a variety of vegetables.

These are just a few examples of the many delicious Chinese dishes out there. Enjoy exploring the world of Chinese cuisine!
```