[BUG] Inconsistent inference and evaluation results of the XLNET-CLM even on the training set!

Bug description

Hello, I followed the example and successfully trained an XLNet-CLM model on my custom dataset. However, I noticed that while the model performs well on the validation set with trainer.evaluate() (even achieving 90% recall@5), I encountered many errors when using trainer.predict() for inference, falling far short of the expected performance.
So, I conducted an experiment:
I took a portion of the training set data and input it into both functions. I used sequence[:] for evaluate() and sequence[:-1] for predict():

=========data for eval===============
   session_id                                      item_id-list
0           1          [26, 26, 26, 26, 4, 4, 4, 4, 4, 4, 4, 4]
1           2       [7, 43, 35, 3, 3, 7, 29, 35, 35, 111, 5, 9]
2           3       [74, 7, 74, 7, 7, 110, 32, 67, 4, 4, 17, 7]
4           5   [56, 25, 25, 25, 25, 25, 25, 23, 34, 4, 19, 43]
5           6  [270, 41, 41, 41, 41, 7, 43, 34, 78, 38, 71, 23]
6           7        [74, 28, 28, 5, 5, 24, 9, 5, 5, 59, 4, 91]
=========data for infer===============
   session_id                                  item_id-list
0           1         [26, 26, 26, 26, 4, 4, 4, 4, 4, 4, 4]
1           2      [7, 43, 35, 3, 3, 7, 29, 35, 35, 111, 5]
2           3      [74, 7, 74, 7, 7, 110, 32, 67, 4, 4, 17]
4           5   [56, 25, 25, 25, 25, 25, 25, 23, 34, 4, 19]
5           6  [270, 41, 41, 41, 41, 7, 43, 34, 78, 38, 71]
6           7        [74, 28, 28, 5, 5, 24, 9, 5, 5, 59, 4]
=========labels:===============
0     4
1     9
2     7
4    43
5    23
6    91
Name: item_id-list, dtype: int64

Ideally, the predictions from both functions should be similar, but it seems the inference results is significantly worse:

=========inference===============
PredictionOutput(predictions=(array([[ 4, 19,  7,  5, 11],
       [ 5, 15, 23, 22,  7],
       [17,  7, 30, 70, 15],
       [19,  4, 11,  5, 15],
       [ 7,  4, 79, 34,  6],
       [ 4,  7, 19, 11,  5]]), array([[7.677282 , 5.3613596, 5.00848  , 4.6888046, 4.319791 ],
       [7.15173  , 5.7525525, 5.726646 , 4.9717607, 4.903692 ],
       [7.843027 , 7.171323 , 5.413306 , 5.378131 , 5.3367157],
       [6.309387 , 5.833148 , 5.6351004, 4.9878273, 4.4770455],
       [7.6406755, 5.9271154, 5.91128  , 5.8692527, 5.589204 ],
       [7.7423315, 5.4810033, 5.0784097, 4.7432585, 4.603503 ]],
      dtype=float32)), label_ids=None, metrics={'test_runtime': 0.792, 'test_samples_per_second': 7.576, 'test_steps_per_second': 3.788})
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 120.82it/s]
=========evaluation===============
PredictionOutput(predictions=(array([[  4,  12,  19,   7,  36],
       [  9,  12,  10,  33,  49],
       [  7,  34,   4,  30,  79],
       [ 43, 110, 129,   6, 126],
       [ 23,  34,  15,  56,  42],
       [ 91,  59, 102, 121,  83]]), array([[18.594198 , 10.1901245, 10.138629 ,  9.487766 ,  9.470682 ],
       [18.095396 , 11.1290245, 10.906204 , 10.659245 , 10.633735 ],
       [16.824923 ,  9.672426 ,  9.347423 ,  8.993546 ,  8.8174095],
       [15.070214 , 10.285906 ,  9.3903475,  8.776609 ,  8.753696 ],
       [16.2281   , 10.282413 , 10.155615 ,  9.619689 ,  9.48603  ],
       [14.316702 , 10.41172  ,  9.787606 ,  9.7606325,  9.695224 ]],
      dtype=float32)), label_ids=array([ 4,  9,  7, 43, 23, 91]), metrics={'eval_/next-item/ndcg_at_5': 1.0, 'eval_/next-item/ndcg_at_10': 1.0, 'eval_/next-item/recall_at_5': 1.0, 'eval_/next-item/recall_at_10': 1.0, 'eval_/next-item/avg_precision_at_5': 1.0, 'eval_/next-item/avg_precision_at_10': 1.0, 'eval_/loss': 0.02894706465303898, 'eval_runtime': 0.0639, 'eval_samples_per_second': 93.926, 'eval_steps_per_second': 46.963})

Please note that I conducted this experiment on the training set. The outputs from evaluation are expected, but the inference not. I'm curious to know why this is happening. Thanks!

Here is my code that generates the outputs above:

  tr_model.load_state_dict(torch.load("tmp/checkpoint-450/pytorch_model.bin"))
  tr_model.eval()

  args = tr.trainer.T4RecTrainingArguments(
          output_dir="tmp",
          per_device_eval_batch_size=2,
          max_sequence_length=30,
          fp16=True,
      )

  trainer = tr.Trainer(
      model=tr_model,
      args=args,
      schema=schema,
      compute_metrics=True,
      )

  trainer.test_dataset_or_path = 'data/preproc_sessions_by_day_3827/1/train_truncate_for_infer.parquet'
  trainer.eval_dataset_or_path = 'data/preproc_sessions_by_day_3827/1/train_truncate_for_eval.parquet'
  trainer.args.predict_top_k = 5

  prediction = trainer.predict(trainer.test_dataset_or_path)
  print("=========inference===============")
  print(prediction)
  
  # a small monkey patch to output the predictions, not only the metrics
  tr.Trainer.evaluate = evaluate_change_output
  prediction = trainer.evaluate()
  print("=========evaluation===============")
  print(prediction)

Environment details

• Transformers4Rec version: 23.06
• Platform: linux
• Python version: 3.10
• Huggingface Transformers version: 4.33.2
• PyTorch version (GPU?):
• Tensorflow version (GPU?):

Start by comparing the trainer.predict() and trainer.evaluate() calls, including their train_truncate_for_infer.parquet and train_truncate_for_eval.parquet inputs, with the reported Transformers4Rec 23.06, Transformers 4.33.2, and PyTorch environment. Reproduce the differing top-k outputs on the training data and determine whether the discrepancy is caused by preprocessing, labels, or the two trainer paths; done means the cause and expected behavior are documented or corrected.