Flant-t5-large Deepspeed OVERFLOW! issues + bad outputs after trlx training

### 🐛 Describe the bug

Hi, I'm trying to use `ilql` training on custom data with `flan-t5-large` and `flan-t5-xl` models to fine-tune them using RLHF and `gpt-j-6B` as a reward model.

1. I have completed the 1st `sft` step for the both `flan-t5` models with a custom training script and a custom dataset.
-> The model is working. I can load it and use it for inference.

2. I have completed the 2nd step, getting `gpt-j` reward_model checkpoints. The model was also fine-tuned in a first `sft` step like the `flan-t5` ones. So, now I get the pytorch_model.bin of this new reward model ready to be used.
I also used custom data with `prompt`, `chosen`, and `rejected` things like mentioned in the [blog post](https://wandb.ai/carperai/summarize_RLHF/reports/Implementing-RLHF-Learning-to-Summarize-with-trlX--VmlldzozMzAwODM2#comparison-dataset).

3. I'm here 👋

I'm using this script from your examples: https://github.com/CarperAI/trlx/blob/main/examples/summarize_rlhf/ilql_summarize_t5.py

So I'm trying to improve the `flan-t5` models' performances using RLHF, but I face problems when it comes to this step:

Deepspeed keeps telling me there is an `OVERFLOW!` issue, and it needs to scale down the loss. And after 8-9 loss scale attempts, the training crashes.

So I trained using suggested models/datasets from the example script to see how it goes on your side by default. And there are also `OVERFLOW!` issues, but after 4 loss scale attempts, it runs well.


- Is my reward model broken?
- Is my dataset too tiny to get something from RLHF?
- Is my parameters bad? (Using the one used in the demo script)
- Could it come from the fact my initial `flan-t5` models were trained with `bf16` or `fp32`?

Any hint would be so much appreciated 🤗

### Which trlX version are you using?

`trlx` main branch

### Additional system and package information

GCP Pytorch 1.13.1, CUDA 11.6, 2xA100 40GBs, transformers>=4.26.1, deepspeed>=0.8.2, accelerate>=0.16.0