[Bug] Internvl2-8B模型量化后推理速度减慢

### Checklist

- [ ] 1. I have searched related issues but cannot get the expected help.
- [ ] 2. The bug has not been fixed in the latest version.
- [ ] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

### Describe the bug

我使用lmdeploy将Internvl2-8B模型用awq量化成INT4，用同一个query对量化前后模型进行推理，量化前`generate_token_len=155`，推理耗时4s；量化后`generate_token_len=134`，推理耗时29s，慢了7倍，这是正常的吗？

### Reproduction

```
from lmdeploy import pipeline, TurbomindEngineConfig, GenerationConfig
import time
import os

gen_config = GenerationConfig(
max_new_tokens=1024,
top_p=1.0,
temperature=0.2,
top_k=5,
repetition_penalty=1.0
)

context_len = 4096
# model_path = "internvl2-8b/v0-20241010-111326/checkpoint-5600-merged-awq-int4/"
# backend_config = TurbomindEngineConfig(model_format='awq', session_len=context_len)

model_path = "internvl2-8b/v0-20241010-111326/checkpoint-5600-merged/"
backend_config = TurbomindEngineConfig(session_len=context_len)

pipe = pipeline(model_path, backend_config=backend_config)

prompt = [
{
'role': 'user',
'content': [
{'type': 'text', 'text': '描述图片内容'},
{'type': 'image_url', 'image_url': {'url': f'{url}'}}
]
}
]

start_time = time.time()
answer = ""
for item in pipe.stream_infer(prompt, gen_config=gen_config):
answer += item.text
print(answer)
print(item.generate_token_len)

print(time.time()-start_time)
os.system("nvidia-smi")
```

### Environment

```Shell
sys.platform: linux
Python: 3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0]
CUDA available: True
MUSA available: False
numpy_random_seed: 2147483648
GPU 0: NVIDIA H20
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.4, V12.4.99
GCC: gcc (GCC) 10.2.1 20200825 (Alibaba 10.2.1-3 2.17)
PyTorch: 2.4.0
PyTorch compiling details: PyTorch built with:
- GCC 10.2
- C++ Version: 201703
- Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67)
- OpenMP 201511 (a.k.a. OpenMP 4.5)
- NNPACK is enabled
- CPU capability usage: AVX512
- CUDA Runtime 12.4
- NVCC architecture flags: -gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_89,code=sm_89;-gencode;arch=compute_90,code=sm_90;-gencode;arch=compute_90,code=compute_90
- CuDNN 90.0 (built against CUDA 12.3)
- Build settings: BUILD_TYPE=Release, CUDA_VERSION=12.4, CUDNN_VERSION=9.0.0, CXX_COMPILER=/bin/c++, CXX_FLAGS=-D_GLIBCXX_USE_CXX11_ABI=0 -std=c++17 -fno-tree-vectorize -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.4.0, USE_CUDA=ON, USE_CUDNN=1, USE_CUSPARSELT=OFF, USE_EIGEN_FOR_BLAS=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=OFF, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF,

TorchVision: 0.19.0a0+48b1edf
LMDeploy: 0.6.1+
transformers: 4.45.2
gradio: 5.0.1
fastapi: 0.115.0
pydantic: 2.9.2
triton: 3.0.0
NVIDIA Topology:
GPU0 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 NIC8 NIC9 NIC10 NIC11 NIC12 NIC13 NIC14 NIC15 NIC16 NIC17 NIC18 NIC19 NIC20 NIC21 NIC22 NIC23 NIC24 NIC25 NIC26 NIC27 NIC28 NIC29 NIC30 NIC31 NIC32 NIC33 NIC34 NIC35 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X SYS SYS SYS SYS SYS SYS SYS SYS PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYSSYS SYS SYS SYS SYS SYS SYS SYS PIX N/A
NIC0 SYS X PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYSSYS SYS SYS SYS SYS SYS SYS PIX SYS
NIC1 SYS PIX X PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYSSYS SYS SYS SYS SYS SYS SYS PIX SYS
NIC2 SYS PIX PIX X PIX PIX PIX PIX PIX SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYSSYS SYS SYS SYS SYS SYS SYS PIX SYS
NIC3 SYS PIX PIX PIX X PIX PIX PIX PIX SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYSSYS SYS SYS SYS SYS SYS SYS PIX SYS
NIC4 SYS PIX PIX PIX PIX X PIX PIX PIX SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYSSYS SYS SYS SYS SYS SYS SYS PIX SYS
NIC5 SYS PIX PIX PIX PIX PIX X PIX PIX SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYSSYS SYS SYS SYS SYS SYS SYS PIX SYS
NIC6 SYS PIX PIX PIX PIX PIX PIX X PIX SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYSSYS SYS SYS SYS SYS SYS SYS PIX SYS
NIC7 SYS PIX PIX PIX PIX PIX PIX PIX X SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYSSYS SYS SYS SYS SYS SYS SYS PIX SYS
NIC8 PIX SYS SYS SYS SYS SYS SYS SYS SYS X PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYSSYS SYS SYS SYS SYS SYS SYS SYS PIX
NIC9 PIX SYS SYS SYS SYS SYS SYS SYS SYS PIX X PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYSSYS SYS SYS SYS SYS SYS SYS SYS PIX
NIC10 PIX SYS SYS SYS SYS SYS SYS SYS SYS PIX PIX X PIX PIX PIX PIX PIX SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYSSYS SYS SYS SYS SYS SYS SYS SYS PIX
NIC11 PIX SYS SYS SYS SYS SYS SYS SYS SYS PIX PIX PIX X PIX PIX PIX PIX SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYSSYS SYS SYS SYS SYS SYS SYS SYS PIX
NIC12 PIX SYS SYS SYS SYS SYS SYS SYS SYS PIX PIX PIX PIX X PIX PIX PIX SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYSSYS SYS SYS SYS SYS SYS SYS SYS PIX
NIC13 PIX SYS SYS SYS SYS SYS SYS SYS SYS PIX PIX PIX PIX PIX X PIX PIX SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYSSYS SYS SYS SYS SYS SYS SYS SYS PIX
NIC14 PIX SYS SYS SYS SYS SYS SYS SYS SYS PIX PIX PIX PIX PIX PIX X PIX SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYSSYS SYS SYS SYS SYS SYS SYS SYS PIX
NIC15 PIX SYS SYS SYS SYS SYS SYS SYS SYS PIX PIX PIX PIX PIX PIX PIX X SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYSSYS SYS SYS SYS SYS SYS SYS SYS PIX
NIC16 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS X PIX PIX PIX PIX PIX PIX PIX SYS SYS SYSSYS SYS SYS SYS SYS PIX SYS SYS SYS
NIC17 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PIX X PIX PIX PIX PIX PIX PIX SYS SYS SYSSYS SYS SYS SYS SYS PIX SYS SYS SYS
NIC18 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PIX PIX X PIX PIX PIX PIX PIX SYS SYS SYSSYS SYS SYS SYS SYS PIX SYS SYS SYS
NIC19 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PIX PIX PIX X PIX PIX PIX PIX SYS SYS SYSSYS SYS SYS SYS SYS PIX SYS SYS SYS
NIC20 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PIX PIX PIX PIX X PIX PIX PIX SYS SYS SYSSYS SYS SYS SYS SYS PIX SYS SYS SYS
NIC21 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PIX PIX PIX PIX PIX X PIX PIX SYS SYS SYSSYS SYS SYS SYS SYS PIX SYS SYS SYS
NIC22 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PIX PIX PIX PIX PIX PIX X PIX SYS SYS SYSSYS SYS SYS SYS SYS PIX SYS SYS SYS
NIC23 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PIX PIX PIX PIX PIX PIX PIX X SYS SYS SYSSYS SYS SYS SYS SYS PIX SYS SYS SYS
NIC24 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS X PIX PIXPIX PIX PIX PIX PIX SYS PIX SYS SYS
NIC25 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PIX X PIXPIX PIX PIX PIX PIX SYS PIX SYS SYS
NIC26 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PIX PIX X PIX PIX PIX PIX PIX SYS PIX SYS SYS
NIC27 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PIX PIX PIX X PIX PIX PIX PIX SYS PIX SYS SYS
NIC28 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PIX PIX PIXPIX X PIX PIX PIX SYS PIX SYS SYS
NIC29 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PIX PIX PIXPIX PIX X PIX PIX SYS PIX SYS SYS
NIC30 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PIX PIX PIXPIX PIX PIX X PIX SYS PIX SYS SYS
NIC31 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PIX PIX PIXPIX PIX PIX PIX X SYS PIX SYS SYS
NIC32 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYSSYS SYS SYS SYS SYS X SYS SYS SYS
NIC33 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PIX PIX PIXPIX PIX PIX PIX PIX SYS X SYS SYS
NIC34 SYS PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYSSYS SYS SYS SYS SYS SYS SYS X SYS
NIC35 PIX SYS SYS SYS SYS SYS SYS SYS SYS PIX PIX PIX PIX PIX PIX PIX PIX SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYSSYS SYS SYS SYS SYS SYS SYS SYS X

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks

NIC Legend:

NIC0: mlx5_0
NIC1: mlx5_1
NIC2: mlx5_2
NIC3: mlx5_3
NIC4: mlx5_4
NIC5: mlx5_5
NIC6: mlx5_6
NIC7: mlx5_7
NIC8: mlx5_8
NIC9: mlx5_9
NIC10: mlx5_10
NIC11: mlx5_11
NIC12: mlx5_12
NIC13: mlx5_13
NIC14: mlx5_14
NIC15: mlx5_15
NIC16: mlx5_16
NIC17: mlx5_17
NIC18: mlx5_18
NIC19: mlx5_19
NIC20: mlx5_20
NIC21: mlx5_21
NIC22: mlx5_22
NIC23: mlx5_23
NIC24: mlx5_24
NIC25: mlx5_25
NIC26: mlx5_26
NIC27: mlx5_27
NIC28: mlx5_28
NIC29: mlx5_29
NIC30: mlx5_30
NIC31: mlx5_31
NIC32: mlx5_bond_0
NIC33: mlx5_bond_1
NIC34: mlx5_bond_2
NIC35: mlx5_bond_3
```

### Error traceback

_No response_