Debugging sudden gradient overflows?

I'm implementing a BERT-like model in FP16 and training works well for the first few thousand iterations, but both with dynamic loss scale (scaling factor goes all the way down to 1) and static factor of 128, at some point I reach a gradient overflow error at each iteration that is inescapable. The loss up to that point is a reasonable and small scalar (around 4) and it kind of happens out of nowhere. I haven't been able to reproduce the exact cause.

What are some good ways to track down the source of this error, and what are some possible causes you might be familiar with? I'm still on the old API but from what I understand switching to the new one shouldn't fix this problem one way or another.

Thanks!

No files, tests, or concrete entry points are named. Start by reducing the BERT-like FP16 training case to a reproducible example and inspect the old API's loss-scaling path around the first overflow. The issue does not provide enough detail to define a repository change or completion criterion.