使用KV cache（int8或int4）量化internvl-v1.5后，显存反而增加了

### Checklist

- [X] 1. I have searched related issues but cannot get the expected help.
- [ ] 2. The bug has not been fixed in the latest version.

### Describe the bug

使用KV cache（int8或int4）量化internvl-v1.5后，GPU显存占用增加到了73G，直接用Transformers加载只有50G左右。这个是怎么回事？该怎么修复？谢谢！
量化代码如下：

```
engine_config = TurbomindEngineConfig(quant_policy=4)
pipe = pipeline('./models/InternVL-Chat-V1-5',
backend_config=TurbomindEngineConfig(quant_policy=4, session_len=4096))
```

### Reproduction

```
engine_config = TurbomindEngineConfig(quant_policy=4)
pipe = pipeline('./models/InternVL-Chat-V1-5',
backend_config=TurbomindEngineConfig(quant_policy=4, session_len=4096))
```

### Environment

```Shell
...
```

### Error traceback

_No response_