[Bug] pytorch backend 's precision points loss 1.0-2.5 points between main code and v0.6.1 on some models.

### Checklist

- [ ] 1. I have searched related issues but cannot get the expected help.
- [ ] 2. The bug has not been fixed in the latest version.
- [ ] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

### Describe the bug

Precision points of pytorch backend loss between main code and v0.6.1.

### Reproduction

1. install opencompass
2. copy .github/scripts/eval_chat_config.py, add following code on end of config
```
models=[pytorch_internlm2_5_7b_chat, pytorch_llama3_8b_instruct, pytorch_qwen1_5_moe_2_7b_chat]
datasets=[*mmlu_datasets]
```
3. run opencompass .github/scripts/eval_chat_config.py to do evaluation

### Environment

```Shell
v0.6.2
```

### Error traceback

_No response_