[Bug] 为什么minicpm-v2_5 使用awq int4量化后速度比fp16慢三倍

### Checklist

- [x] 1. I have searched related issues but cannot get the expected help.
- [x] 2. The bug has not been fixed in the latest version.
- [x] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

### Describe the bug

lmdeploy fp16 推理时间2541 ms

量化后int4推理8221 ms

### Reproduction

量化命令：
```
lmdeploy lite auto_awq \
$HF_MODEL \
--calib-samples 128 \
--calib-seqlen 200 \
--w-bits 4 \
--w-group-size 128 \
--batch-size 128 \
--search-scale False \
--work-dir $WORK_DIR
```
推理代码
```python
def lmdeploy_infer_awq_model():
from PIL import Image
from lmdeploy import pipeline, TurbomindEngineConfig, GenerationConfig
gen_config = GenerationConfig(temperature=0.7, top_k=100, top_p=0.8, repetition_penalty=1.05)
engine_config = TurbomindEngineConfig(model_format='awq')
model_name = './MiniCPM-Llama3-V-2_5-4bit-a100-40g'
pipe = pipeline(model_name, backend_config=engine_config)
# image = Image.open('demo.png').convert('RGB')
from lmdeploy.vl import load_image
image = load_image('demo.png')
response = pipe(('详细描述图片内容', image), gen_config=gen_config)
print(response)
t1 = time.time()
for _ in range(loops):
response = pipe(('详细描述图片内容', image), gen_config=gen_config)
t2 = time.time()
print(f'Inference {loops} times takes {t2 - t1} seconds')
print(f"latency: {(t2 - t1) / loops} seconds")
```

### Environment

```Shell
cuda11.8, lmdeploy 0.5.2.post
```

### Error traceback

_No response_