SIGFPE / Floating Point Exception using Prodigy

I wanted to post here that running the SDXL training script with the latest branch and training crashed consistently with SIGFPE (floating point exception).

I'm using an Nvidia T4, CUDA 12 and Torch 2.10. Among other settings, I used FP16 mixed precision training and the Prodigy optimizer. Using Prodigy specifically caused the floating point crash.

After trying many different things, I finally was able to resolve the issue by installing Torch 2.9.1.

Start with the SDXL training script using FP16 mixed precision, Prodigy, an Nvidia T4, CUDA 12, and Torch 2.10, and reproduce the SIGFPE while comparing Torch 2.9.1. Done means identifying the compatibility cause and documenting or testing a fix for this configuration.

Written by the indexing model from the issue text.