Ascend v0.7.2.post1，对serving api测速，第一次测试吞吐少40%+

### Checklist

- [x] 1. I have searched related issues but cannot get the expected help.
- [x] 2. The bug has not been fixed in the latest version.
- [x] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

### Describe the bug

ascend卡部署QwQ-32B，使用官方制作的ascend镜像

先用一些短文本warmup，然后使用1k个更长的文本对api server进行测速，1k prompt跑多次，第一次跑吞吐少很多

### Reproduction

serving
```bash
lmdeploy serve api_server \
--backend pytorch \
--device ascend \
--server-port 23333 \
--tp 8 \
--dtype bfloat16 \
--chat-template qwen2d5 \
--model-name QwQ-32B \
/cache/hf_models/QwQ-32B/
```
首先，warmup
```bash
MODEL_NAME="QwQ-32B"
TOKENIZER_PATH=/cache/hf_models/QwQ-32B/
SEED=42

for N in 1 64 256 512
do
echo "warmup with N=${N}"
python profile_restful_api.py \
--host localhost \
--port 23333 \
--backend lmdeploy \
--dataset-name random \
--dataset-path /cache/ShareGPT_V3_unfiltered_cleaned_split.json \
--random-input-len 256 \
--random-output-len 128 \
--random-range-ratio 0.5 \
--model ${MODEL_NAME} \
--tokenizer ${TOKENIZER_PATH} \
--seed ${SEED} \
--num-prompts ${N}
done
```
然后，使用1k个prompt测试，运行多次
```bash
MODEL_NAME="QwQ-32B"
TOKENIZER_PATH=/cache/hf_models/QwQ-32B/
SEED=42
NUM_PROMPTS=1000

python profile_restful_api.py \
--host localhost \
--port 23333 \
--backend lmdeploy \
--dataset-name random \
--dataset-path /cache/ShareGPT_V3_unfiltered_cleaned_split.json \
--random-input-len 1024 \
--random-output-len 1024 \
--random-range-ratio 0.5 \
--model ${MODEL_NAME} \
--tokenizer ${TOKENIZER_PATH} \
--seed ${SEED} \
--num-prompts ${NUM_PROMPTS}
```

第一次测试的结果output throughput是\~1550 tok/s，后续每次运行是\~2200 tok/s。

这个有什么方式可以优化吗？

### Environment

```Shell
sys.platform: linux
Python: 3.10.5 (main, Mar 24 2025, 07:28:13) [GCC 9.4.0]
CUDA available: False
MUSA available: False
numpy_random_seed: 2147483648
GCC: gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
PyTorch: 2.3.1
PyTorch compiling details: PyTorch built with:
- GCC 10.2
- C++ Version: 201703
- Intel(R) MKL-DNN v3.3.6 (Git Hash 86e6af5974177e513fd3fee58425e1063e7f1361)
- OpenMP 201511 (a.k.a. OpenMP 4.5)
- LAPACK is enabled (usually provided by MKL)
- NNPACK is enabled
- CPU capability usage: NO AVX
- Build settings: BLAS_INFO=open, BUILD_TYPE=Release, CXX_COMPILER=/opt/rh/devtoolset-10/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=open, TORCH_VERSION=2.3.1, USE_CUDA=OFF, USE_CUDNN=OFF, USE_CUSPARSELT=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF,

TorchVision: 0.18.1
LMDeploy: 0.7.2.post1+
transformers: 4.50.0
gradio: Not Found
fastapi: 0.115.12
pydantic: 2.10.6
triton: Not Found
```

### Error traceback

```Shell

```