[BUG] A bug while fine-tuning the model by iteratively training and evaluating using a sliding time window

Bug description

I find out that there is no label in valid.parquet.

Steps/Code to reproduce bug

While I m running this code:
start_time_window_index = 1
final_time_window_index = 4
for time_index in range(start_time_window_index, final_time_window_index):
# Set data
time_index_train = time_index
time_index_eval = time_index + 1
train_paths = glob.glob(os.path.join(OUTPUT_DIR, f"{time_index_train}/train.parquet"))
eval_paths = glob.glob(os.path.join(OUTPUT_DIR, f"{time_index_eval}/valid.parquet"))
# Train on day related to time_index
print('*'20)
print("Launch training for day %s are:" %time_index)
print(''20 + '\n')
trainer.train_dataset_or_path = train_paths
trainer.reset_lr_scheduler()
trainer.train()
trainer.state.global_step +=1
# Evaluate on the following day
trainer.eval_dataset_or_path = eval_paths
train_metrics = trainer.evaluate(metric_key_prefix='eval')
print(''20)
print("Eval results for day %s are:\t" %time_index_eval)
print('\n' + ''*20 + '\n')
for key in sorted(train_metrics.keys()):
print(" %s = %s" % (key, str(train_metrics[key])))
wipe_memory()

the error appear:

Launch training for day 1 are:

/usr/local/lib/python3.10/dist-packages/transformers/optimization.py:391: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set no_deprecation_warning=True to disable this warning
warnings.warn(
{'train_runtime': 4.0234, 'train_samples_per_second': 3817.691, 'train_steps_per_second': 14.913, 'train_loss': 10.525657145182292, 'epoch': 60.0}
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 60/60 [00:04<00:00, 14.92it/s]
TrainOutput(global_step=60, training_loss=10.525657145182292, metrics={'train_runtime': 4.0234, 'train_samples_per_second': 3817.691, 'train_steps_per_second': 14.913, 'total_flos': 0.0, 'train_loss': 10.525657145182292})
Traceback (most recent call last):
File "", line 17, in
File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 2932, in evaluate
output = eval_loop(
File "/usr/local/lib/python3.10/dist-packages/transformers4rec/torch/trainer.py", line 515, in evaluation_loop
metrics_results_detailed = model.calculate_metrics(preds, labels)
File "/usr/local/lib/python3.10/dist-packages/transformers4rec/torch/model/base.py", line 616, in calculate_metrics
head.calculate_metrics(
File "/usr/local/lib/python3.10/dist-packages/transformers4rec/torch/model/base.py", line 453, in calculate_metrics
task.calculate_metrics(
File "/usr/local/lib/python3.10/dist-packages/transformers4rec/torch/model/prediction_task.py", line 489, in calculate_metrics
result = metric(predictions, targets)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torchmetrics/metric.py", line 301, in forward
self._forward_cache = self._forward_full_state_update(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torchmetrics/metric.py", line 316, in _forward_full_state_update
self.update(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torchmetrics/metric.py", line 465, in wrapped_func
update(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/transformers4rec/torch/ranking_metric.py", line 56, in update
metric = self._metric(
File "/usr/local/lib/python3.10/dist-packages/transformers4rec/torch/ranking_metric.py", line 137, in _metric
if rel_indices.shape[0] > 0:
IndexError: tuple index out of range

Expected behavior

I expected there have label for evaluation

Environment details

• Transformers4Rec version: 23.12
• Platform:Docker
• Python version:3.10
• Huggingface Transformers version:4.27.1
• PyTorch version (GPU?):2.1.0a0+4136153
• Tensorflow version (GPU?):

Additional context

Start by reproducing the sliding-window loop with Transformers4Rec 23.12 and inspect the generated valid.parquet files for the evaluation label. Then trace the evaluation path through transformers4rec/torch/trainer.py, model/base.py, prediction_task.py, and ranking_metric.py. Done means evaluation receives valid labels and completes without the IndexError.