[Bug] vl pipeline triggle cudaMemcpyAsync ERROR illegal memory access

### Checklist

- [X] 1. I have searched related issues but cannot get the expected help.
- [X] 2. The bug has not been fixed in the latest version.

### Describe the bug

we use vl_pipeline following [Batch prompts inference](https://github.com/InternLM/lmdeploy/blob/main/docs/en/inference/vl_pipeline.md#batch-prompts-inference)

we got this error:
```
terminate called after throwing an instance of 'std::runtime_error'
what(): [TM][ERROR] CUDA runtime error: an illegal memory access was encountered /lmdeploy/src/turbomind/models/llama/LlamaBatch.h:134
```

here is the code in LlamaBatch.h:
```
// analogs to `std::copy_n`
template
U* Copy(const U* src, size_t count, U* dst)
{
check_cuda_error(cudaMemcpyAsync(dst, src, sizeof(U) * count, cudaMemcpyDefault, stream_));
return dst += count;
}
```

### Reproduction

### command to reproduce: CUDA_VISIBLE_DEVICES=0 python test.py
``` test.py
from lmdeploy import pipeline, ChatTemplateConfig
from lmdeploy.messages import VisonConfig
from lmdeploy.vl import load_image
pipe = pipeline('liuhaotian/llava-v1.6-vicuna-7b', log_level='INFO',vision_config=VisonConfig(max_batch_size=1))
image_urls=['https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/demo/resources/det.jpg' for i in range(40)]
prompt_list = [('describe this image', load_image(img_url)) for img_url in image_urls]
for i in range(100):
pipe(prompt_list)
```

We found that the smaller the setting of max_batch_size, the easier it is to reproduce this issue; if the size of the prompt_list is equal to max_batch_size, the issue basically does not occur.
We also found that it cannot be reproduced on A100

### Environment

```Shell
## hardware
nvidia L40(sm 89)

## python env
conda create -n lmdeploy python=3.10

### conda list -n lmdeploy

# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge
_openmp_mutex 4.5 2_gnu
accelerate 0.31.0 pypi_0 pypi
addict 2.4.0 pypi_0 pypi
annotated-types 0.7.0 pypi_0 pypi
anyio 4.4.0 pypi_0 pypi
asttokens 2.4.1 pypi_0 pypi
bzip2 1.0.8 hd590300_5
ca-certificates 2024.2.2 hbcca054_0
certifi 2024.6.2 pypi_0 pypi
charset-normalizer 3.3.2 pypi_0 pypi
click 8.1.7 pypi_0 pypi
comm 0.2.2 pypi_0 pypi
debugpy 1.8.1 pypi_0 pypi
decorator 5.1.1 pypi_0 pypi
dnspython 2.6.1 pypi_0 pypi
einops 0.8.0 pypi_0 pypi
email-validator 2.1.2 pypi_0 pypi
exceptiongroup 1.2.1 pypi_0 pypi
executing 2.0.1 pypi_0 pypi
fastapi 0.111.0 pypi_0 pypi
fastapi-cli 0.0.4 pypi_0 pypi
filelock 3.15.1 pypi_0 pypi
fire 0.6.0 pypi_0 pypi
fsspec 2024.6.0 pypi_0 pypi
h11 0.14.0 pypi_0 pypi
httpcore 1.0.5 pypi_0 pypi
httptools 0.6.1 pypi_0 pypi
httpx 0.27.0 pypi_0 pypi
huggingface-hub 0.23.4 pypi_0 pypi
idna 3.7 pypi_0 pypi
importlib-metadata 7.1.0 pypi_0 pypi
ipykernel 6.29.4 pypi_0 pypi
ipython 8.25.0 pypi_0 pypi
jedi 0.19.1 pypi_0 pypi
jinja2 3.1.4 pypi_0 pypi
jupyter-client 8.6.2 pypi_0 pypi
jupyter-core 5.7.2 pypi_0 pypi
ld_impl_linux-64 2.40 h41732ed_0
libffi 3.4.2 h7f98852_5
libgcc-ng 13.2.0 h807b86a_5
libgomp 13.2.0 h807b86a_5
libnsl 2.0.1 hd590300_0
libsqlite 3.45.3 h2797004_0
libuuid 2.38.1 h0b41bf4_0
libxcrypt 4.4.36 hd590300_1
libzlib 1.2.13 hd590300_5
llava 1.2.2.post1 pypi_0 pypi
lmdeploy 0.4.2 pypi_0 pypi
markdown-it-py 3.0.0 pypi_0 pypi
markupsafe 2.1.5 pypi_0 pypi
matplotlib-inline 0.1.7 pypi_0 pypi
mdurl 0.1.2 pypi_0 pypi
mmengine-lite 0.10.4 pypi_0 pypi
mpmath 1.3.0 pypi_0 pypi
ncurses 6.4.20240210 h59595ed_0
nest-asyncio 1.6.0 pypi_0 pypi
networkx 3.3 pypi_0 pypi
numpy 1.26.4 pypi_0 pypi
nvidia-cublas-cu12 12.1.3.1 pypi_0 pypi
nvidia-cuda-cupti-cu12 12.1.105 pypi_0 pypi
nvidia-cuda-nvrtc-cu12 12.1.105 pypi_0 pypi
nvidia-cuda-runtime-cu12 12.1.105 pypi_0 pypi
nvidia-cudnn-cu12 8.9.2.26 pypi_0 pypi
nvidia-cufft-cu12 11.0.2.54 pypi_0 pypi
nvidia-curand-cu12 10.3.2.106 pypi_0 pypi
nvidia-cusolver-cu12 11.4.5.107 pypi_0 pypi
nvidia-cusparse-cu12 12.1.0.106 pypi_0 pypi
nvidia-nccl-cu12 2.20.5 pypi_0 pypi
nvidia-nvjitlink-cu12 12.5.40 pypi_0 pypi
nvidia-nvtx-cu12 12.1.105 pypi_0 pypi
openssl 3.2.1 hd590300_1
orjson 3.10.5 pypi_0 pypi
packaging 24.1 pypi_0 pypi
parso 0.8.4 pypi_0 pypi
peft 0.9.0 pypi_0 pypi
pexpect 4.9.0 pypi_0 pypi
pillow 10.3.0 pypi_0 pypi
pip 24.0 pyhd8ed1ab_0
platformdirs 4.2.2 pypi_0 pypi
prompt-toolkit 3.0.47 pypi_0 pypi
protobuf 5.27.1 pypi_0 pypi
psutil 5.9.8 pypi_0 pypi
ptyprocess 0.7.0 pypi_0 pypi
pure-eval 0.2.2 pypi_0 pypi
pydantic 2.7.4 pypi_0 pypi
pydantic-core 2.18.4 pypi_0 pypi
pygments 2.18.0 pypi_0 pypi
pynvml 11.5.0 pypi_0 pypi
python 3.10.14 hd12c33a_0_cpython
python-dateutil 2.9.0.post0 pypi_0 pypi
python-dotenv 1.0.1 pypi_0 pypi
python-multipart 0.0.9 pypi_0 pypi
pyyaml 6.0.1 pypi_0 pypi
pyzmq 26.0.3 pypi_0 pypi
readline 8.2 h8228510_1
regex 2024.5.15 pypi_0 pypi
requests 2.32.3 pypi_0 pypi
rich 13.7.1 pypi_0 pypi
safetensors 0.4.3 pypi_0 pypi
sentencepiece 0.2.0 pypi_0 pypi
setuptools 69.5.1 pyhd8ed1ab_0
shellingham 1.5.4 pypi_0 pypi
shortuuid 1.0.13 pypi_0 pypi
six 1.16.0 pypi_0 pypi
sniffio 1.3.1 pypi_0 pypi
stack-data 0.6.3 pypi_0 pypi
starlette 0.37.2 pypi_0 pypi
sympy 1.12.1 pypi_0 pypi
termcolor 2.4.0 pypi_0 pypi
tiktoken 0.7.0 pypi_0 pypi
tk 8.6.13 noxft_h4845f30_101
tokenizers 0.19.1 pypi_0 pypi
tomli 2.0.1 pypi_0 pypi
torch 2.3.1 pypi_0 pypi
torchvision 0.17.2 pypi_0 pypi
tornado 6.4.1 pypi_0 pypi
tqdm 4.66.4 pypi_0 pypi
traitlets 5.14.3 pypi_0 pypi
transformers 4.41.2 pypi_0 pypi
triton 2.3.1 pypi_0 pypi
typer 0.12.3 pypi_0 pypi
typing-extensions 4.12.2 pypi_0 pypi
tzdata 2024a h0c530f3_0
ujson 5.10.0 pypi_0 pypi
urllib3 2.2.2 pypi_0 pypi
uvicorn 0.30.1 pypi_0 pypi
uvloop 0.19.0 pypi_0 pypi
watchfiles 0.22.0 pypi_0 pypi
wcwidth 0.2.13 pypi_0 pypi
websockets 12.0 pypi_0 pypi
wheel 0.43.0 pyhd8ed1ab_1
xz 5.2.6 h166bdaf_0
yapf 0.40.2 pypi_0 pypi
zipp 3.19.2 pypi_0 pypi
```
```