Loss Increasing after few epochs

Hi @rnyak , @benfred @gabrielspmoreira @oliverholworthy , I'm training the t4rec model on custom data, but the loss is not decreasing after few epochs, instead it started increasing. Basically the loss started from 13.67 and after training for few epoch it get decreased to 6.43 and then it started increasing,

I'm not sure what can be done to improve the loss more.

Here are my params:

params = {
    'batch_size': 512,
    'lr': 0.0005,
    'lr_scheduler': 'cosine',
    'num_train_epochs': 1,
    'using_test': True,
    'using_type': False,
    'bl_shuffle': True,
    'masking': 'mlm',
    'd_model': 256,
    'n_head': 32,
    'n_layer': 3,
    'proj_num': 1,
    'act_mlp': 'None',
    'item_correction': False,
    'neg_factor': 4,
    'label_smoothing': 0.0,
    'temperature': 1.5734215681668653,
    'remove_false_neg': True,
    'item_correction_factor': 0.04152252077012748,
    'transformer_dropout': 0.05096800263401626,
    'mlm_probability': 0.35044384745899415,
    'top20': True,
    'loss_types': True,
    'loss_types_type': 'Simple',
    'multi_task_emb': 0,
    'mt_num_layers': 1,
    'use_tanh': False,
    'seq_len': 20,
    'split': 0
}

Any suggesstion would be very helpful. Thanks in Advance!!

Originally posted by @alan-ai-learner in https://github.com/NVIDIA-Merlin/Transformers4Rec/issues/493#issuecomment-1471570279

No source file, test, or specific entry point is identified in the issue. Start by reproducing the training run with the listed parameters on the custom data, then inspect the training and loss behavior around the point where the loss begins increasing. Done means identifying the cause and documenting a verified configuration or code fix.