【Program hangs with no output.】

I am conducting the instruction tuning of llama3_llava using the script on my own dataset
`NPROC_PER_NODE=${GPU_NUM} xtuner train llava_llama3_8b_instruct_full_clip_vit_large_p14_336_lora_e1_gpu8_finetune --deepspeed deepspeed_zero3_offload --seed 1024`. After the following output, the program stops outputting but is still running:
```
- mmengine - INFO - Iter(train) [ 10/23076] lr: 1.3034e-07 eta: 3 days, 3:30:35 time: 11.7851 data_time: 0.0298 memory: 15547 loss: nan
- mmengine - INFO - Iter(train) [ 20/23076] lr: 2.7506e-07 eta: 3 days, 5:46:56 time: 12.5050 data_time: 0.0199 memory: 9964 loss: nan
```

This state has been ongoing for 2 hours. What could be the possible cause for this?