How to choose the best training model

Hello,

In the training process, we save one model every 5000 iterations. Besides training loss, are there any criteria to help choose the best model? Because I think even if training loss is low, the performance is probably not good.
For example, I trained two models. One is model_400k iteration, the other is model_500k iteration. But the actual decoded content 400k's model is more reasonable than 500k's.

Thanks a lot！