[Bug] 4-bit KV cache results in very low performance while 8-bit KV cache is almost lossless

### Checklist

- [X] 1. I have searched related issues but cannot get the expected help.
- [X] 2. The bug has not been fixed in the latest version.
- [X] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

### Describe the bug

Hi thanks for the library! I am trying to run qwen2.5-math-7B on 4090. It seems the 8-bit kv cache runs quite well without losing performance (on my internal math dataset), while 4bit makes it give 0% accuracy.

### Reproduction

(see above)

### Environment

```Shell
docker
```

### Error traceback

_No response_