lmdeploy量化后速度比浮点速度慢的问题？？？InternVL2.5-4B

### Checklist

- [ ] 1. I have searched related issues but cannot get the expected help.
- [ ] 2. The bug has not been fixed in the latest version.
- [ ] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

### Describe the bug

使用lmdeploy中的autoawq在A800和A100上量化后，模型速度变慢，int8也同样出现该问题，模型是InternVL2_5-4B-MPO，有大佬熟悉吗？

### Reproduction

A800/A100上量化测试结果如下：

浮点模型耗时：0.5s / 0.64s

int4耗时： 0.62s/0.78s（重新量化, A100上结论一致）

int8耗时：0.94s / 1.1s

fp8耗时：1.45s / 1.45s

int4量化命令：
export HF_MODEL=OpenGVLab/InternVL2_5-4B-MPO
export WORK_DIR=OpenGVLab/InternVL2_5-4B-MPO-4bit

lmdeploy lite auto_awq \
$HF_MODEL \
--calib-dataset 'ptb' \
--calib-samples 128 \
--calib-seqlen 2048 \
--w-bits 4 \
--dtype 'float16' \
--w-group-size 128 \
--batch-size 1 \
--work-dir $WORK_DIR

int8量化命令：

export HF_MODEL=OpenGVLab/InternVL2_5-4B-MPO
export WORK_DIR=OpenGVLab/InternVL2_5-4B-MPO-int8
#
lmdeploy lite smooth_quant \
$HF_MODEL \
--dtype 'bfloat16' \
--quant-dtype 'int8' \
--work-dir $WORK_DIR

int4耗时测试方法：

engine_config = TurbomindEngineConfig(model_format='awq', session_len=8192)
gen_config = GenerationConfig(top_k=40, top_p=0.8, temperature=0.3)
internvl_quan_model_path = path
pipe = pipeline(internvl_quan_model_path, backend_config=engine_config)
t1 = time.time()
response = pipe((prompt, images), gen_config = gen_config)
t2 = time.time()

int8和浮点类似；

### Environment

```Shell
lmdeploy 0.7.1
```

### Error traceback

```Shell

```