reward collapse still occurred in spg_mix

I did run spg_mix in gsm8k without changing the source code, but a reward collapse still occurred. Why is that?

When I see the logs, the per_seq_logps tensor becomes nan:

```
WARNING: NaN/inf detected in per_seq_logps!
per_seq_logps: tensor([[nan, nan, nan, nan, nan, nan]], device='cuda:0', grad_fn=)
```

And the model just keeps generating !!!!!!!!!!!!!!!!!!!!!!!!!
```
Response:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
```

Is there something fundamental error in SPG? Also, I found that the strict_format_reward_func and soft_format_reward_func are both wrong. Have you really evaluated the format reward functions?