[Bug] CUDA error: an illegal memory access was encountered. Vicuna results wrong

### Checklist

- [ ] 1. I have searched related issues but cannot get the expected help.
- [ ] 2. The bug has not been fixed in the latest version.
- [ ] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

### Describe the bug

The result of vicuna got wrong since #2810. Using cuda graph causes the program crash. Using eager-mode causes wrong result.
The wrong result:
```
Response(text='ican help you with anything?', generate_token_len=7, input_token_len=41, session_id=59, finish_reason='stop', token_ids=[2185, 1371, 366, 411, 3099, 29973, 2], logprobs=None, index=58), Response(text=". I'm a language model called Vicuna, and I'm a language model developed by Large Model Systems Organization (LMSYS).", generate_token_len=32, input_token_len=44, session_id=60, finish_reason='stop', token_ids=[29889, 306, 29915, 29885, 263, 4086, 1904, 2000, 13423, 4347, 29892, 322, 306, 29915, 29885, 263, 4086, 1904, 8906, 491, 8218, 479, 8125, 23985, 9205, 2133, 313, 29931, 4345, 21554, 467, 2], logprobs=None, index=59)]
```
The expected result:
```
Response(text='我是一个名为 Vicuna 的大型语言模型，我是基于 GPT-3.5 架构的。我是一个大型语言模型，可以理解和生成自然语言。我可以回答各种问题，提供各种信息，并可以进行自然语言处理任务。', generate_token_len=96, input_token_len=44, session_id=60, finish_reason='stop', token_ids=[30672, 30392, 30287, 30502, 30548, 30573, 13423, 4347, 29871, 30210, 30257, 30883, 31505, 31243, 31382, 30883, 30214, 30672, 30392, 31359, 30909, 402, 7982, 29899, 29941, 29889, 29945, 29871, 233, 161, 185, 31901, 30210, 30267, 30672, 30392, 30287, 30502, 30257, 30883, 31505, 31243, 31382, 30883, 30214, 30682, 30651, 30687, 31201, 30503, 30486, 30494, 30688, 31516, 31505, 31243, 30267, 30672, 30682, 30651, 30742, 234, 176, 151, 232, 147, 135, 31893, 31658, 31596, 30214, 31302, 231, 193, 158, 232, 147, 135, 31893, 30689, 31021, 30214, 31666, 30682, 30651, 31174, 30448, 30688, 31516, 31505, 31243, 31548, 30687, 31450, 31358, 30267], logprobs=None, index=59)]
```

### Reproduction

```python
from lmdeploy import pipeline
from lmdeploy.vl import load_image
from lmdeploy.messages import GenerationConfig, PytorchEngineConfig, TurbomindEngineConfig
model = '/mnt/140/vicuna/vicuna-7b-v1.3'
if __name__ == '__main__':
pipe = pipeline(model, backend_config=PytorchEngineConfig(tp=1, eager_mode=False, max_batch_size=256), log_level='INFO')
for res in pipe.stream_infer('write a long poem'):
print(res.text, end='')
response = pipe(['hi','who are you？']*30)
print(response)
```

### Environment

```Shell
sys.platform: linux
Python: 3.10.10 (main, Mar 21 2023, 18:45:11) [GCC 11.2.0]
CUDA available: True
MUSA available: False
numpy_random_seed: 2147483648
GPU 0: NVIDIA A100-SXM4-80GB
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.4, V12.4.131
GCC: gcc (GCC) 9.3.1 20200408 (Red Hat 9.3.1-2)
PyTorch: 2.4.0+cu118
PyTorch compiling details: PyTorch built with:
- GCC 9.3
- C++ Version: 201703
- Intel(R) oneAPI Math Kernel Library Version 2022.2-Product Build 20220804 for Intel(R) 64 architecture applications
- Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67)
- OpenMP 201511 (a.k.a. OpenMP 4.5)
- LAPACK is enabled (usually provided by MKL)
- NNPACK is enabled
- CPU capability usage: AVX2
- CUDA Runtime 11.8
- NVCC architecture flags: -gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_90,code=sm_90
- CuDNN 90.1
- Magma 2.6.1
- Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.8, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.4.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=1, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF,

TorchVision: 0.19.0+cu118
LMDeploy: 0.6.5+ac509e8
transformers: 4.37.0
gradio: 5.6.0
fastapi: 0.115.5
pydantic: 2.10.1
triton: 3.0.0
NVIDIA Topology:
GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X NV12 NV12 NV12 NV12 NV12 NV12 NV12 SYS SYS SYS 48-63 3 N/A
GPU1 NV12 X NV12 NV12 NV12 NV12 NV12 NV12 SYS SYS SYS 48-63 3 N/A
GPU2 NV12 NV12 X NV12 NV12 NV12 NV12 NV12 PXB PXB SYS 16-31 1 N/A
GPU3 NV12 NV12 NV12 X NV12 NV12 NV12 NV12 PXB PXB SYS 16-31 1 N/A
GPU4 NV12 NV12 NV12 NV12 X NV12 NV12 NV12 SYS SYS PXB 96-111 6 N/A
GPU5 NV12 NV12 NV12 NV12 NV12 X NV12 NV12 SYS SYS PXB 96-111 6 N/A
GPU6 NV12 NV12 NV12 NV12 NV12 NV12 X NV12 SYS SYS SYS 64-79 4 N/A
GPU7 NV12 NV12 NV12 NV12 NV12 NV12 NV12 X SYS SYS SYS 64-79 4 N/A
NIC0 SYS SYS PXB PXB SYS SYS SYS SYS X PXB SYS
NIC1 SYS SYS PXB PXB SYS SYS SYS SYS PXB X SYS
NIC2 SYS SYS SYS SYS PXB PXB SYS SYS SYS SYS X

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks

NIC Legend:

NIC0: mlx5_0
NIC1: mlx5_1
NIC2: mlx5_bond_0
```

### Error traceback

```Shell
miniconda3/lib/python3.10/site-packages/torch/cuda/streams.py", line 206, in query
return super().query()
RuntimeError: CUDA error: an illegal memory access was encountered
```