When should I stop training?

Hi there,

Thanks for your work, InsightFace. I have a couple of questions regarding the training ArcFace model with my custom dataset.

**General information**
- Dataset of about 40M images with 800K unique identities
- Trained with DDP on 3 GPUs with batch-size of 25 images per GPU
- IR-SE ResNet50 model from scratch
- He-Xavier Normal weights initialization in ArcFace Head layer
- Trained already 4 epochs

As a rule of thumb, I am using **cross-entropy loss** as the loss function in the FC layer. While training, I also validate the embeddings (512,1) of the model, and the performance as following **acc: +99.8, prec: +99.7, recall: +99.7**. I validate model on 240K matches (120K person A and person B, 120K person A and person A). While achieving these results, my training CE loss is around ~15.
I suppose that this loss value (~15), in general, is quite large, but in this situation it is more like related to the fact that it is summed along all 800K FC nodes, so accumulating in higher loss.

So my question, based on information provided above, when should I stop training ? I tend to believe that relatively high CE loss with value 15 is not bad, since it represents the robustness and generalization potential of the main model and FC layer, the latter one is not that important in future use of the model.

Any advice and help will be much appreciated.