[Bug] internvl2-2b使用awq量化后，推理速度基本上没有提升，精度还掉点

### Checklist

- [ ] 1. I have searched related issues but cannot get the expected help.
- [ ] 2. The bug has not been fixed in the latest version.
- [ ] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

### Describe the bug

internvl2-2b使用awq w4a16量化后，推理速度基本上没有提升，精度还掉点，请问是什么原因，没有走到LMDeploy专门实现的w4a16的pattern吗？

### Reproduction

机器：NVIDIA L40，CUDA 12.1
量化命令：
```sh
lmdeploy lite auto_awq \
$HF_MODEL \
--calib-samples 128 \
--calib-seqlen 1024 \
--w-bits 4 \
--w-group-size 128 \
--batch-size 8 \
--search-scale True \
--work-dir $WORK_DIR
```

### Environment

```Shell
lmdeploy 0.5.2.post1
```

### Error traceback

_No response_