Clarification of the arrangement of the Tensor Benchmark Generator

Hello Authors. I would like to check with you about my code as I am generating a benchmark using Tensor Benchmark Generator from my custom dataset which is not on Avalanche.

I have arranged my dataset with different classes from each training experience [0, 1, 2, 3, 4]. For example, for each experience i, the input data (clientTrainX[i]) consists of 2 different classes. After training in the first experience is finished, I want to test the experience with the testing set to observe its accuracy. Then, the next training in the second experience started, and after it is finished, I want to test it with the same testing set to observe the change in accuracy. The process repeats for the next experience until the last experience.

You can find my code here for reference.
`
generic_scenario = tensors_benchmark(
train_tensors=[(client_trainX[0], client_trainy[0]),
(client_trainX[1], client_trainy[1]),
(client_trainX[2], client_trainy[2]),
(client_trainX[3], client_trainy[3]),
(client_trainX[4], client_trainy[4])
],
test_tensors=[(client_testX, client_testy)],
task_labels=[0, 1, 2, 3, 4], # Task label of each train exp
complete_test_set_only= True
)
`

My problem is when I run the training code. During the evaluation, it shows that it performs the evaluation only on experience 0 (Task 0) from test stream in every experience until the end (as shown below). Does this indicate that I have made any mistakes in generating my benchmark above? Is there any problem with "complete_test_set_only= True" or task_labels setting? I also noticed that the forgetting value is strange too. I would be glad if you could help me with the syntax according to my purpose above, please? I really appreciate any help you can provide.

`
Starting experiment...
Start of experience: 0
Current Classes: [0, 1]
-- >> Start of training phase << --
100%|██████████| 16/16 [00:00<00:00, 139.12it/s]
Epoch 0 ended.
Loss_Epoch/train_phase/train_stream/Task000 = 1.6401
Top1_Acc_Epoch/train_phase/train_stream/Task000 = 0.4900
100%|██████████| 16/16 [00:00<00:00, 163.15it/s]
Epoch 1 ended.
Loss_Epoch/train_phase/train_stream/Task000 = 1.4517
Top1_Acc_Epoch/train_phase/train_stream/Task000 = 0.5913
100%|██████████| 16/16 [00:00<00:00, 158.95it/s]
Epoch 2 ended.
Loss_Epoch/train_phase/train_stream/Task000 = 1.3243
Top1_Acc_Epoch/train_phase/train_stream/Task000 = 0.7625
100%|██████████| 16/16 [00:00<00:00, 180.46it/s]
Epoch 3 ended.
Loss_Epoch/train_phase/train_stream/Task000 = 1.2159
Top1_Acc_Epoch/train_phase/train_stream/Task000 = 0.8638
100%|██████████| 16/16 [00:00<00:00, 160.48it/s]
Epoch 4 ended.
Loss_Epoch/train_phase/train_stream/Task000 = 1.1629
Top1_Acc_Epoch/train_phase/train_stream/Task000 = 0.8962
100%|██████████| 16/16 [00:00<00:00, 176.06it/s]
Epoch 5 ended.
Loss_Epoch/train_phase/train_stream/Task000 = 1.1249
Top1_Acc_Epoch/train_phase/train_stream/Task000 = 0.9325
100%|██████████| 16/16 [00:00<00:00, 148.04it/s]
Epoch 6 ended.
Loss_Epoch/train_phase/train_stream/Task000 = 1.0976
Top1_Acc_Epoch/train_phase/train_stream/Task000 = 0.9525
100%|██████████| 16/16 [00:00<00:00, 172.41it/s]
Epoch 7 ended.
Loss_Epoch/train_phase/train_stream/Task000 = 1.0900
Top1_Acc_Epoch/train_phase/train_stream/Task000 = 0.9575
100%|██████████| 16/16 [00:00<00:00, 161.61it/s]
Epoch 8 ended.
Loss_Epoch/train_phase/train_stream/Task000 = 1.0843
Top1_Acc_Epoch/train_phase/train_stream/Task000 = 0.9650
100%|██████████| 16/16 [00:00<00:00, 191.73it/s]
Epoch 9 ended.
Loss_Epoch/train_phase/train_stream/Task000 = 1.0789
Top1_Acc_Epoch/train_phase/train_stream/Task000 = 0.9700
100%|██████████| 16/16 [00:00<00:00, 155.79it/s]
Epoch 10 ended.
Loss_Epoch/train_phase/train_stream/Task000 = 1.0775
Top1_Acc_Epoch/train_phase/train_stream/Task000 = 0.9688
100%|██████████| 16/16 [00:00<00:00, 181.80it/s]
Epoch 11 ended.
Loss_Epoch/train_phase/train_stream/Task000 = 1.0758
Top1_Acc_Epoch/train_phase/train_stream/Task000 = 0.9712
100%|██████████| 16/16 [00:00<00:00, 176.78it/s]
Epoch 12 ended.
Loss_Epoch/train_phase/train_stream/Task000 = 1.0705
Top1_Acc_Epoch/train_phase/train_stream/Task000 = 0.9762
100%|██████████| 16/16 [00:00<00:00, 169.40it/s]
Epoch 13 ended.
Loss_Epoch/train_phase/train_stream/Task000 = 1.0704
Top1_Acc_Epoch/train_phase/train_stream/Task000 = 0.9775
100%|██████████| 16/16 [00:00<00:00, 159.02it/s]
Epoch 14 ended.
Loss_Epoch/train_phase/train_stream/Task000 = 1.0685
Top1_Acc_Epoch/train_phase/train_stream/Task000 = 0.9800
100%|██████████| 16/16 [00:00<00:00, 154.34it/s]
Epoch 15 ended.
Loss_Epoch/train_phase/train_stream/Task000 = 1.0648
Top1_Acc_Epoch/train_phase/train_stream/Task000 = 0.9800
100%|██████████| 16/16 [00:00<00:00, 169.27it/s]
Epoch 16 ended.
Loss_Epoch/train_phase/train_stream/Task000 = 1.0633
Top1_Acc_Epoch/train_phase/train_stream/Task000 = 0.9825
100%|██████████| 16/16 [00:00<00:00, 154.55it/s]
Epoch 17 ended.
Loss_Epoch/train_phase/train_stream/Task000 = 1.0632
Top1_Acc_Epoch/train_phase/train_stream/Task000 = 0.9825
100%|██████████| 16/16 [00:00<00:00, 150.20it/s]
Epoch 18 ended.
Loss_Epoch/train_phase/train_stream/Task000 = 1.0609
Top1_Acc_Epoch/train_phase/train_stream/Task000 = 0.9838
100%|██████████| 16/16 [00:00<00:00, 81.93it/s]
Epoch 19 ended.
Loss_Epoch/train_phase/train_stream/Task000 = 1.0603
MeanScores/train_phase/train_stream =
MeanScores/train_phase/train_stream/new_classes = 0.9828
Top1_Acc_Epoch/train_phase/train_stream/Task000 = 0.9850
-- >> End of training phase << --
Training completed
Computing accuracy on the whole test set
-- >> Start of eval phase << --
**-- Starting eval on experience 0 (Task 0) from test stream --**
100%|██████████| 1/1 [00:00<00:00, 26.65it/s]
> Eval on experience 0 (Task 0) from test stream ended.
Loss_Exp/eval_phase/test_stream/Task000/Exp000 = 1.7277
Top1_Acc_Exp/eval_phase/test_stream/Task000/Exp000 = 0.3133
-- >> End of eval phase << --
ConfusionMatrix_Stream/eval_phase/test_stream =
Loss_Stream/eval_phase/test_stream/Task000 = 1.7277
MeanScores/eval_phase/test_stream =
MeanScores/eval_phase/test_stream/new_classes = 0.9379
MeanScores/eval_phase/test_stream/new_old_diff = 0.9379
MeanScores/eval_phase/test_stream/old_classes = 0.0000
StreamForgetting/eval_phase/test_stream = 0.0000
Top1_Acc_Stream/eval_phase/test_stream/Task000 = 0.3133
Top1_ClassAcc_Stream/eval_phase/test_stream/Task000/0 = 0.9000
Top1_ClassAcc_Stream/eval_phase/test_stream/Task000/1 = 0.9800
Top1_ClassAcc_Stream/eval_phase/test_stream/Task000/2 = 0.0000
Top1_ClassAcc_Stream/eval_phase/test_stream/Task000/3 = 0.0000
Top1_ClassAcc_Stream/eval_phase/test_stream/Task000/4 = 0.0000
Top1_ClassAcc_Stream/eval_phase/test_stream/Task000/5 = 0.0000
Start of experience: 1
Current Classes: [2]
-- >> Start of training phase << --
100%|██████████| 8/8 [00:00<00:00, 81.21it/s]
Epoch 0 ended.
Loss_Epoch/train_phase/train_stream/Task001 = 1.5966
Top1_Acc_Epoch/train_phase/train_stream/Task001 = 0.4444
100%|██████████| 8/8 [00:00<00:00, 102.56it/s]
Epoch 1 ended.
Loss_Epoch/train_phase/train_stream/Task001 = 1.5963
Top1_Acc_Epoch/train_phase/train_stream/Task001 = 0.4444
100%|██████████| 8/8 [00:00<00:00, 81.50it/s]
Epoch 2 ended.
Loss_Epoch/train_phase/train_stream/Task001 = 1.5953
Top1_Acc_Epoch/train_phase/train_stream/Task001 = 0.4444
100%|██████████| 8/8 [00:00<00:00, 82.41it/s]
Epoch 3 ended.
Loss_Epoch/train_phase/train_stream/Task001 = 1.5904
Top1_Acc_Epoch/train_phase/train_stream/Task001 = 0.4444
100%|██████████| 8/8 [00:00<00:00, 78.79it/s]
Epoch 4 ended.
Loss_Epoch/train_phase/train_stream/Task001 = 1.4317
Top1_Acc_Epoch/train_phase/train_stream/Task001 = 0.6056
100%|██████████| 8/8 [00:00<00:00, 82.84it/s]
Epoch 5 ended.
Loss_Epoch/train_phase/train_stream/Task001 = 1.2526
Top1_Acc_Epoch/train_phase/train_stream/Task001 = 0.8292
100%|██████████| 8/8 [00:00<00:00, 75.76it/s]
Epoch 6 ended.
Loss_Epoch/train_phase/train_stream/Task001 = 1.1506
Top1_Acc_Epoch/train_phase/train_stream/Task001 = 0.9292
100%|██████████| 8/8 [00:00<00:00, 82.41it/s]
Epoch 7 ended.
Loss_Epoch/train_phase/train_stream/Task001 = 1.0941
Top1_Acc_Epoch/train_phase/train_stream/Task001 = 0.9764
100%|██████████| 8/8 [00:00<00:00, 76.06it/s]
Epoch 8 ended.
Loss_Epoch/train_phase/train_stream/Task001 = 1.0664
Top1_Acc_Epoch/train_phase/train_stream/Task001 = 0.9861
100%|██████████| 8/8 [00:00<00:00, 84.77it/s]
Epoch 9 ended.
Loss_Epoch/train_phase/train_stream/Task001 = 1.0555
Top1_Acc_Epoch/train_phase/train_stream/Task001 = 0.9972
100%|██████████| 8/8 [00:00<00:00, 75.42it/s]
Epoch 10 ended.
Loss_Epoch/train_phase/train_stream/Task001 = 1.0540
Top1_Acc_Epoch/train_phase/train_stream/Task001 = 0.9958
100%|██████████| 8/8 [00:00<00:00, 88.51it/s]
Epoch 11 ended.
Loss_Epoch/train_phase/train_stream/Task001 = 1.0521
Top1_Acc_Epoch/train_phase/train_stream/Task001 = 0.9958
100%|██████████| 8/8 [00:00<00:00, 79.04it/s]
Epoch 12 ended.
Loss_Epoch/train_phase/train_stream/Task001 = 1.0504
Top1_Acc_Epoch/train_phase/train_stream/Task001 = 0.9972
100%|██████████| 8/8 [00:00<00:00, 82.52it/s]
Epoch 13 ended.
Loss_Epoch/train_phase/train_stream/Task001 = 1.0517
Top1_Acc_Epoch/train_phase/train_stream/Task001 = 0.9958
100%|██████████| 8/8 [00:00<00:00, 85.78it/s]
Epoch 14 ended.
Loss_Epoch/train_phase/train_stream/Task001 = 1.0477
Top1_Acc_Epoch/train_phase/train_stream/Task001 = 1.0000
100%|██████████| 8/8 [00:00<00:00, 78.86it/s]
Epoch 15 ended.
Loss_Epoch/train_phase/train_stream/Task001 = 1.0479
Top1_Acc_Epoch/train_phase/train_stream/Task001 = 0.9986
100%|██████████| 8/8 [00:00<00:00, 88.23it/s]
Epoch 16 ended.
Loss_Epoch/train_phase/train_stream/Task001 = 1.0472
Top1_Acc_Epoch/train_phase/train_stream/Task001 = 1.0000
100%|██████████| 8/8 [00:00<00:00, 79.61it/s]
Epoch 17 ended.
Loss_Epoch/train_phase/train_stream/Task001 = 1.0465
Top1_Acc_Epoch/train_phase/train_stream/Task001 = 1.0000
100%|██████████| 8/8 [00:00<00:00, 90.30it/s]
Epoch 18 ended.
Loss_Epoch/train_phase/train_stream/Task001 = 1.0464
Top1_Acc_Epoch/train_phase/train_stream/Task001 = 1.0000
100%|██████████| 8/8 [00:00<00:00, 42.32it/s]
Epoch 19 ended.
Loss_Epoch/train_phase/train_stream/Task001 = 1.0460
MeanScores/train_phase/train_stream =
MeanScores/train_phase/train_stream/new_classes = 0.9995
MeanScores/train_phase/train_stream/new_old_diff = 0.0012
MeanScores/train_phase/train_stream/old_classes = 0.9983
Top1_Acc_Epoch/train_phase/train_stream/Task001 = 1.0000
-- >> End of training phase << --
Training completed
Computing accuracy on the whole test set
-- >> Start of eval phase << --
**-- Starting eval on experience 0 (Task 0) from test stream --**
100%|██████████| 1/1 [00:00<00:00, 28.95it/s]
> Eval on experience 0 (Task 0) from test stream ended.
ExperienceForgetting/eval_phase/test_stream/Task000/Exp000 = 0.0250
Loss_Exp/eval_phase/test_stream/Task000/Exp000 = 1.7407
Top1_Acc_Exp/eval_phase/test_stream/Task000/Exp000 = 0.2883
-- >> End of eval phase << --
ConfusionMatrix_Stream/eval_phase/test_stream =
Loss_Stream/eval_phase/test_stream/Task000 = 1.7407
MeanScores/eval_phase/test_stream =
MeanScores/eval_phase/test_stream/new_classes = 0.9277
MeanScores/eval_phase/test_stream/new_old_diff = 0.7683
MeanScores/eval_phase/test_stream/old_classes = 0.1594
StreamForgetting/eval_phase/test_stream = 0.0250
Top1_Acc_Stream/eval_phase/test_stream/Task000 = 0.2883
Top1_ClassAcc_Stream/eval_phase/test_stream/Task000/0 = 0.4300
Top1_ClassAcc_Stream/eval_phase/test_stream/Task000/1 = 0.3600
Top1_ClassAcc_Stream/eval_phase/test_stream/Task000/2 = 0.9400
Top1_ClassAcc_Stream/eval_phase/test_stream/Task000/3 = 0.0000
Top1_ClassAcc_Stream/eval_phase/test_stream/Task000/4 = 0.0000
Top1_ClassAcc_Stream/eval_phase/test_stream/Task000/5 = 0.0000
Start of experience: 2
Current Classes: [3]
-- >> Start of training phase << --
100%|██████████| 8/8 [00:00<00:00, 74.47it/s]
Epoch 0 ended.
Loss_Epoch/train_phase/train_stream/Task002 = 1.5842
Top1_Acc_Epoch/train_phase/train_stream/Task002 = 0.4444
100%|██████████| 8/8 [00:00<00:00, 80.63it/s]
Epoch 1 ended.
Loss_Epoch/train_phase/train_stream/Task002 = 1.5742
Top1_Acc_Epoch/train_phase/train_stream/Task002 = 0.4444
100%|██████████| 8/8 [00:00<00:00, 76.83it/s]
Epoch 2 ended.
Loss_Epoch/train_phase/train_stream/Task002 = 1.5531
Top1_Acc_Epoch/train_phase/train_stream/Task002 = 0.4444
100%|██████████| 8/8 [00:00<00:00, 82.01it/s]
Epoch 3 ended.
Loss_Epoch/train_phase/train_stream/Task002 = 1.4912
Top1_Acc_Epoch/train_phase/train_stream/Task002 = 0.4444
100%|██████████| 8/8 [00:00<00:00, 78.39it/s]
Epoch 4 ended.
Loss_Epoch/train_phase/train_stream/Task002 = 1.4562
Top1_Acc_Epoch/train_phase/train_stream/Task002 = 0.6361
100%|██████████| 8/8 [00:00<00:00, 85.55it/s]
Epoch 5 ended.
Loss_Epoch/train_phase/train_stream/Task002 = 1.3658
Top1_Acc_Epoch/train_phase/train_stream/Task002 = 0.8694
100%|██████████| 8/8 [00:00<00:00, 79.83it/s]
Epoch 6 ended.
Loss_Epoch/train_phase/train_stream/Task002 = 1.1311
Top1_Acc_Epoch/train_phase/train_stream/Task002 = 0.9750
100%|██████████| 8/8 [00:00<00:00, 86.34it/s]
Epoch 7 ended.
Loss_Epoch/train_phase/train_stream/Task002 = 1.0664
Top1_Acc_Epoch/train_phase/train_stream/Task002 = 0.9875
100%|██████████| 8/8 [00:00<00:00, 82.05it/s]
Epoch 8 ended.
Loss_Epoch/train_phase/train_stream/Task002 = 1.0512
Top1_Acc_Epoch/train_phase/train_stream/Task002 = 0.9972
100%|██████████| 8/8 [00:00<00:00, 77.16it/s]
Epoch 9 ended.
Loss_Epoch/train_phase/train_stream/Task002 = 1.0509
Top1_Acc_Epoch/train_phase/train_stream/Task002 = 0.9944
100%|██████████| 8/8 [00:00<00:00, 83.67it/s]
Epoch 10 ended.
Loss_Epoch/train_phase/train_stream/Task002 = 1.0497
Top1_Acc_Epoch/train_phase/train_stream/Task002 = 0.9944
100%|██████████| 8/8 [00:00<00:00, 85.10it/s]
Epoch 11 ended.
Loss_Epoch/train_phase/train_stream/Task002 = 1.0478
Top1_Acc_Epoch/train_phase/train_stream/Task002 = 0.9972
100%|██████████| 8/8 [00:00<00:00, 82.01it/s]
Epoch 12 ended.
Loss_Epoch/train_phase/train_stream/Task002 = 1.0465
Top1_Acc_Epoch/train_phase/train_stream/Task002 = 0.9986
100%|██████████| 8/8 [00:00<00:00, 80.65it/s]
Epoch 13 ended.
Loss_Epoch/train_phase/train_stream/Task002 = 1.0468
Top1_Acc_Epoch/train_phase/train_stream/Task002 = 0.9986
100%|██████████| 8/8 [00:00<00:00, 80.36it/s]
Epoch 14 ended.
Loss_Epoch/train_phase/train_stream/Task002 = 1.0461
Top1_Acc_Epoch/train_phase/train_stream/Task002 = 0.9986
100%|██████████| 8/8 [00:00<00:00, 76.16it/s]
Epoch 15 ended.
Loss_Epoch/train_phase/train_stream/Task002 = 1.0461
Top1_Acc_Epoch/train_phase/train_stream/Task002 = 0.9986
100%|██████████| 8/8 [00:00<00:00, 80.01it/s]
Epoch 16 ended.
Loss_Epoch/train_phase/train_stream/Task002 = 1.0461
Top1_Acc_Epoch/train_phase/train_stream/Task002 = 0.9986
100%|██████████| 8/8 [00:00<00:00, 77.16it/s]
Epoch 17 ended.
Loss_Epoch/train_phase/train_stream/Task002 = 1.0462
Top1_Acc_Epoch/train_phase/train_stream/Task002 = 0.9986
100%|██████████| 8/8 [00:00<00:00, 86.44it/s]
Epoch 18 ended.
Loss_Epoch/train_phase/train_stream/Task002 = 1.0459
Top1_Acc_Epoch/train_phase/train_stream/Task002 = 0.9986
100%|██████████| 8/8 [00:00<00:00, 41.36it/s]
Epoch 19 ended.
Loss_Epoch/train_phase/train_stream/Task002 = 1.0461
MeanScores/train_phase/train_stream =
MeanScores/train_phase/train_stream/new_classes = 0.9972
MeanScores/train_phase/train_stream/new_old_diff = -0.0023
MeanScores/train_phase/train_stream/old_classes = 0.9995
Top1_Acc_Epoch/train_phase/train_stream/Task002 = 0.9986
-- >> End of training phase << --
Training completed
Computing accuracy on the whole test set
-- >> Start of eval phase << --
**-- Starting eval on experience 0 (Task 0) from test stream --**
100%|██████████| 1/1 [00:00<00:00, 27.38it/s]
> Eval on experience 0 (Task 0) from test stream ended.
ExperienceForgetting/eval_phase/test_stream/Task000/Exp000 = -0.0233
Loss_Exp/eval_phase/test_stream/Task000/Exp000 = 1.7012
Top1_Acc_Exp/eval_phase/test_stream/Task000/Exp000 = 0.3367
-- >> End of eval phase << --
ConfusionMatrix_Stream/eval_phase/test_stream =
Loss_Stream/eval_phase/test_stream/Task000 = 1.7012
MeanScores/eval_phase/test_stream =
MeanScores/eval_phase/test_stream/new_classes = 0.9933
MeanScores/eval_phase/test_stream/new_old_diff = 0.7916
MeanScores/eval_phase/test_stream/old_classes = 0.2017
StreamForgetting/eval_phase/test_stream = -0.0233
Top1_Acc_Stream/eval_phase/test_stream/Task000 = 0.3367
Top1_ClassAcc_Stream/eval_phase/test_stream/Task000/0 = 0.1900
Top1_ClassAcc_Stream/eval_phase/test_stream/Task000/1 = 0.1800
Top1_ClassAcc_Stream/eval_phase/test_stream/Task000/2 = 0.6500
Top1_ClassAcc_Stream/eval_phase/test_stream/Task000/3 = 1.0000
Top1_ClassAcc_Stream/eval_phase/test_stream/Task000/4 = 0.0000
Top1_ClassAcc_Stream/eval_phase/test_stream/Task000/5 = 0.0000
Start of experience: 3
Current Classes: [4]
-- >> Start of training phase << --
100%|██████████| 8/8 [00:00<00:00, 77.06it/s]
Epoch 0 ended.
Loss_Epoch/train_phase/train_stream/Task003 = 1.5972
Top1_Acc_Epoch/train_phase/train_stream/Task003 = 0.4444
100%|██████████| 8/8 [00:00<00:00, 81.95it/s]
Epoch 1 ended.
Loss_Epoch/train_phase/train_stream/Task003 = 1.5912
Top1_Acc_Epoch/train_phase/train_stream/Task003 = 0.4431
100%|██████████| 8/8 [00:00<00:00, 72.54it/s]
Epoch 2 ended.
Loss_Epoch/train_phase/train_stream/Task003 = 1.5845
Top1_Acc_Epoch/train_phase/train_stream/Task003 = 0.4444
100%|██████████| 8/8 [00:00<00:00, 84.90it/s]
Epoch 3 ended.
Loss_Epoch/train_phase/train_stream/Task003 = 1.5625
Top1_Acc_Epoch/train_phase/train_stream/Task003 = 0.4444
100%|██████████| 8/8 [00:00<00:00, 74.93it/s]
Epoch 4 ended.
Loss_Epoch/train_phase/train_stream/Task003 = 1.5130
Top1_Acc_Epoch/train_phase/train_stream/Task003 = 0.4444
100%|██████████| 8/8 [00:00<00:00, 82.83it/s]
Epoch 5 ended.
Loss_Epoch/train_phase/train_stream/Task003 = 1.4681
Top1_Acc_Epoch/train_phase/train_stream/Task003 = 0.4500
100%|██████████| 8/8 [00:00<00:00, 77.26it/s]
Epoch 6 ended.
Loss_Epoch/train_phase/train_stream/Task003 = 1.3314
Top1_Acc_Epoch/train_phase/train_stream/Task003 = 0.7694
100%|██████████| 8/8 [00:00<00:00, 77.98it/s]
Epoch 7 ended.
Loss_Epoch/train_phase/train_stream/Task003 = 1.1987
Top1_Acc_Epoch/train_phase/train_stream/Task003 = 0.8528
100%|██████████| 8/8 [00:00<00:00, 80.14it/s]
Epoch 8 ended.
Loss_Epoch/train_phase/train_stream/Task003 = 1.1913
Top1_Acc_Epoch/train_phase/train_stream/Task003 = 0.8528
100%|██████████| 8/8 [00:00<00:00, 83.50it/s]
Epoch 9 ended.
Loss_Epoch/train_phase/train_stream/Task003 = 1.1915
Top1_Acc_Epoch/train_phase/train_stream/Task003 = 0.8528
100%|██████████| 8/8 [00:00<00:00, 79.99it/s]
Epoch 10 ended.
Loss_Epoch/train_phase/train_stream/Task003 = 1.1924
Top1_Acc_Epoch/train_phase/train_stream/Task003 = 0.8514
100%|██████████| 8/8 [00:00<00:00, 81.62it/s]
Epoch 11 ended.
Loss_Epoch/train_phase/train_stream/Task003 = 1.1894
Top1_Acc_Epoch/train_phase/train_stream/Task003 = 0.8542
100%|██████████| 8/8 [00:00<00:00, 70.68it/s]
Epoch 12 ended.
Loss_Epoch/train_phase/train_stream/Task003 = 1.1894
Top1_Acc_Epoch/train_phase/train_stream/Task003 = 0.8542
100%|██████████| 8/8 [00:00<00:00, 75.67it/s]
Epoch 13 ended.
Loss_Epoch/train_phase/train_stream/Task003 = 1.1883
Top1_Acc_Epoch/train_phase/train_stream/Task003 = 0.8556
100%|██████████| 8/8 [00:00<00:00, 77.13it/s]
Epoch 14 ended.
Loss_Epoch/train_phase/train_stream/Task003 = 1.1893
Top1_Acc_Epoch/train_phase/train_stream/Task003 = 0.8542
100%|██████████| 8/8 [00:00<00:00, 83.90it/s]
Epoch 15 ended.
Loss_Epoch/train_phase/train_stream/Task003 = 1.1881
Top1_Acc_Epoch/train_phase/train_stream/Task003 = 0.8556
100%|██████████| 8/8 [00:00<00:00, 76.21it/s]
Epoch 16 ended.
Loss_Epoch/train_phase/train_stream/Task003 = 1.1883
Top1_Acc_Epoch/train_phase/train_stream/Task003 = 0.8556
100%|██████████| 8/8 [00:00<00:00, 80.39it/s]
Epoch 17 ended.
Loss_Epoch/train_phase/train_stream/Task003 = 1.1884
Top1_Acc_Epoch/train_phase/train_stream/Task003 = 0.8556
100%|██████████| 8/8 [00:00<00:00, 75.02it/s]
Epoch 18 ended.
Loss_Epoch/train_phase/train_stream/Task003 = 1.1882
Top1_Acc_Epoch/train_phase/train_stream/Task003 = 0.8556
100%|██████████| 8/8 [00:00<00:00, 46.53it/s]
Epoch 19 ended.
Loss_Epoch/train_phase/train_stream/Task003 = 1.1887
MeanScores/train_phase/train_stream =
MeanScores/train_phase/train_stream/new_classes = 0.9999
MeanScores/train_phase/train_stream/new_old_diff = 0.3264
MeanScores/train_phase/train_stream/old_classes = 0.6735
Top1_Acc_Epoch/train_phase/train_stream/Task003 = 0.8556
-- >> End of training phase << --
Training completed
Computing accuracy on the whole test set
-- >> Start of eval phase << --
**-- Starting eval on experience 0 (Task 0) from test stream --**
100%|██████████| 1/1 [00:00<00:00, 31.25it/s]
> Eval on experience 0 (Task 0) from test stream ended.
ExperienceForgetting/eval_phase/test_stream/Task000/Exp000 = 0.0300
Loss_Exp/eval_phase/test_stream/Task000/Exp000 = 1.7553
Top1_Acc_Exp/eval_phase/test_stream/Task000/Exp000 = 0.2833
-- >> End of eval phase << --
ConfusionMatrix_Stream/eval_phase/test_stream =
Loss_Stream/eval_phase/test_stream/Task000 = 1.7553
MeanScores/eval_phase/test_stream =
MeanScores/eval_phase/test_stream/new_classes = 0.9998
MeanScores/eval_phase/test_stream/new_old_diff = 0.8614
MeanScores/eval_phase/test_stream/old_classes = 0.1384
StreamForgetting/eval_phase/test_stream = 0.0300
Top1_Acc_Stream/eval_phase/test_stream/Task000 = 0.2833
Top1_ClassAcc_Stream/eval_phase/test_stream/Task000/0 = 0.1200
Top1_ClassAcc_Stream/eval_phase/test_stream/Task000/1 = 0.1900
Top1_ClassAcc_Stream/eval_phase/test_stream/Task000/2 = 0.3900
Top1_ClassAcc_Stream/eval_phase/test_stream/Task000/3 = 0.0000
Top1_ClassAcc_Stream/eval_phase/test_stream/Task000/4 = 1.0000
Top1_ClassAcc_Stream/eval_phase/test_stream/Task000/5 = 0.0000
Start of experience: 4
Current Classes: [5]
-- >> Start of training phase << --
100%|██████████| 8/8 [00:00<00:00, 55.13it/s]
Epoch 0 ended.
Loss_Epoch/train_phase/train_stream/Task004 = 1.7089
Top1_Acc_Epoch/train_phase/train_stream/Task004 = 0.3333
100%|██████████| 8/8 [00:00<00:00, 63.44it/s]
Epoch 1 ended.
Loss_Epoch/train_phase/train_stream/Task004 = 1.7048
Top1_Acc_Epoch/train_phase/train_stream/Task004 = 0.3333
100%|██████████| 8/8 [00:00<00:00, 82.84it/s]
Epoch 2 ended.
Loss_Epoch/train_phase/train_stream/Task004 = 1.7087
Top1_Acc_Epoch/train_phase/train_stream/Task004 = 0.3319
100%|██████████| 8/8 [00:00<00:00, 78.12it/s]
Epoch 3 ended.
Loss_Epoch/train_phase/train_stream/Task004 = 1.7053
Top1_Acc_Epoch/train_phase/train_stream/Task004 = 0.3319
100%|██████████| 8/8 [00:00<00:00, 83.76it/s]
Epoch 4 ended.
Loss_Epoch/train_phase/train_stream/Task004 = 1.7010
Top1_Acc_Epoch/train_phase/train_stream/Task004 = 0.3333
100%|██████████| 8/8 [00:00<00:00, 81.82it/s]
Epoch 5 ended.
Loss_Epoch/train_phase/train_stream/Task004 = 1.6569
Top1_Acc_Epoch/train_phase/train_stream/Task004 = 0.3889
100%|██████████| 8/8 [00:00<00:00, 82.76it/s]
Epoch 6 ended.
Loss_Epoch/train_phase/train_stream/Task004 = 1.6371
Top1_Acc_Epoch/train_phase/train_stream/Task004 = 0.4167
100%|██████████| 8/8 [00:00<00:00, 76.48it/s]
Epoch 7 ended.
Loss_Epoch/train_phase/train_stream/Task004 = 1.6258
Top1_Acc_Epoch/train_phase/train_stream/Task004 = 0.4222
100%|██████████| 8/8 [00:00<00:00, 79.50it/s]
Epoch 8 ended.
Loss_Epoch/train_phase/train_stream/Task004 = 1.6151
Top1_Acc_Epoch/train_phase/train_stream/Task004 = 0.4333
100%|██████████| 8/8 [00:00<00:00, 77.58it/s]
Epoch 9 ended.
Loss_Epoch/train_phase/train_stream/Task004 = 1.6105
Top1_Acc_Epoch/train_phase/train_stream/Task004 = 0.4333
100%|██████████| 8/8 [00:00<00:00, 77.85it/s]
Epoch 10 ended.
Loss_Epoch/train_phase/train_stream/Task004 = 1.6030
Top1_Acc_Epoch/train_phase/train_stream/Task004 = 0.4292
100%|██████████| 8/8 [00:00<00:00, 79.58it/s]
Epoch 11 ended.
Loss_Epoch/train_phase/train_stream/Task004 = 1.6001
Top1_Acc_Epoch/train_phase/train_stream/Task004 = 0.4333
100%|██████████| 8/8 [00:00<00:00, 80.20it/s]
Epoch 12 ended.
Loss_Epoch/train_phase/train_stream/Task004 = 1.5992
Top1_Acc_Epoch/train_phase/train_stream/Task004 = 0.4333
100%|██████████| 8/8 [00:00<00:00, 90.36it/s]
Epoch 13 ended.
Loss_Epoch/train_phase/train_stream/Task004 = 1.5986
Top1_Acc_Epoch/train_phase/train_stream/Task004 = 0.4333
100%|██████████| 8/8 [00:00<00:00, 69.73it/s]
Epoch 14 ended.
Loss_Epoch/train_phase/train_stream/Task004 = 1.5944
Top1_Acc_Epoch/train_phase/train_stream/Task004 = 0.4319
100%|██████████| 8/8 [00:00<00:00, 67.54it/s]
Epoch 15 ended.
Loss_Epoch/train_phase/train_stream/Task004 = 1.5926
Top1_Acc_Epoch/train_phase/train_stream/Task004 = 0.4319
100%|██████████| 8/8 [00:00<00:00, 70.96it/s]
Epoch 16 ended.
Loss_Epoch/train_phase/train_stream/Task004 = 1.5857
Top1_Acc_Epoch/train_phase/train_stream/Task004 = 0.4333
100%|██████████| 8/8 [00:00<00:00, 74.30it/s]
Epoch 17 ended.
Loss_Epoch/train_phase/train_stream/Task004 = 1.5831
Top1_Acc_Epoch/train_phase/train_stream/Task004 = 0.4333
100%|██████████| 8/8 [00:00<00:00, 77.28it/s]
Epoch 18 ended.
Loss_Epoch/train_phase/train_stream/Task004 = 1.5803
Top1_Acc_Epoch/train_phase/train_stream/Task004 = 0.4333
100%|██████████| 8/8 [00:00<00:00, 40.64it/s]
Epoch 19 ended.
Loss_Epoch/train_phase/train_stream/Task004 = 1.5765
MeanScores/train_phase/train_stream =
MeanScores/train_phase/train_stream/new_classes = 0.0015
MeanScores/train_phase/train_stream/new_old_diff = -0.9663
MeanScores/train_phase/train_stream/old_classes = 0.9677
Top1_Acc_Epoch/train_phase/train_stream/Task004 = 0.4333
-- >> End of training phase << --
Training completed
Computing accuracy on the whole test set
-- >> Start of eval phase << --
**-- Starting eval on experience 0 (Task 0) from test stream --**
100%|██████████| 1/1 [00:00<00:00, 9.85it/s]
> Eval on experience 0 (Task 0) from test stream ended.
ExperienceForgetting/eval_phase/test_stream/Task000/Exp000 = -0.1300
Loss_Exp/eval_phase/test_stream/Task000/Exp000 = 1.5834
Top1_Acc_Exp/eval_phase/test_stream/Task000/Exp000 = 0.4433
-- >> End of eval phase << --
ConfusionMatrix_Stream/eval_phase/test_stream =
Loss_Stream/eval_phase/test_stream/Task000 = 1.5834
MeanScores/eval_phase/test_stream =
MeanScores/eval_phase/test_stream/new_classes = 0.0129
MeanScores/eval_phase/test_stream/new_old_diff = -0.4997
MeanScores/eval_phase/test_stream/old_classes = 0.5126
StreamForgetting/eval_phase/test_stream = -0.1300
Top1_Acc_Stream/eval_phase/test_stream/Task000 = 0.4433
Top1_ClassAcc_Stream/eval_phase/test_stream/Task000/0 = 0.2500
Top1_ClassAcc_Stream/eval_phase/test_stream/Task000/1 = 0.2100
Top1_ClassAcc_Stream/eval_phase/test_stream/Task000/2 = 0.5500
Top1_ClassAcc_Stream/eval_phase/test_stream/Task000/3 = 0.8200
Top1_ClassAcc_Stream/eval_phase/test_stream/Task000/4 = 0.8300
Top1_ClassAcc_Stream/eval_phase/test_stream/Task000/5 = 0.0000

`

Start from the tensors_benchmark call in the issue and inspect how complete_test_set_only, task_labels, and test_tensors define evaluation experiences. Reproduce the reported evaluation output, then clarify whether the benchmark configuration matches the intended repeated test-set evaluation and document the correct usage or expected behavior.

Written by the indexing model from the issue text.