Training loss extremely noisy during fine-tuning and randomly goes to 0

I'm trying to fine-tune the 6.7B model on my own code dataset. I am running a multinode training with fp32 precision on NVIDIA Tesla V100 GPUs with DeepSpeed ZeRO Stage 3. My training loss seems to randomly fluctuate and go down to zero, I've attached my training loss graph below:

I'm running this on 128 GPUs with a train batch size of 1 per device and no gradient accumulation. I'm not sure what could be the cause of this as I haven't seen this happen with other models with the Llama architecture. Would appreciate any general direction to help debug this, thanks!

The issue names no files, tests, or entry points. Start by reviewing the multinode fine-tuning configuration, including batch size, gradient accumulation, fp32 precision, and DeepSpeed ZeRO Stage 3 settings, then isolate the steps where loss reaches zero. Done means identifying a reproducible cause or narrowing the failure to a specific configuration or training component.

Written by the indexing model from the issue text.