ContinualAI / ContinualAI/avalanche

EarlyStoppingPlugin isn't working properly.

Open
#1,634 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2.1k
Forks
321
PR merge metrics
No merged PRs in 30d

Description

🐛 **Describe the bug**
EarlyStoppingPlugin isn't working properly.
When given a benchmark, code cannot follow the process of supervised training in **metric_name**.

**Result:**
The results of the run are displayed as:

1. When given a task incremental benchmark, EarlyStoppingPlugin couldn't train properly for the last experience.
2. When given a class incremental benchmark, EarlyStoppingPlugin couldn't train properly for the whole experience except the first task.

> I infer that the full_name = names[-1] in class EarlyStoppingPlugin is not a good selector to choose the right metric.

I will show the detail for this issue.

🐜 **To Reproduce**
I will use the example of EarlyStopping to claim this issue. The modification will be highlighted.

`# model
model = MTSimpleMLP() << << using the multi-head model

# Here we show all the MNIST variation we offer in the "classic" benchmarks
if args.mnist_type == "permuted":
benchmark = PermutedMNIST(n_experiences=5, seed=1)
elif args.mnist_type == "rotated":
benchmark = RotatedMNIST(
n_experiences=5, rotations_list=[30, 60, 90, 120, 150], seed=1
)
else:
benchmark = SplitMNIST(n_experiences=5, seed=1, return_task_id=True) << << setting as task increamental learning

# Than we can extract the parallel train and test streams
train_stream = benchmark.train_stream
test_stream = benchmark.test_stream

# Prepare for training & testing
optimizer = SGD(model.parameters(), lr=0.001, momentum=0.9)
criterion = CrossEntropyLoss()

# Continual learning strategy with default logger
cl_strategy = Naive(
model,
optimizer,
criterion,
train_mb_size=32,
train_epochs=100,
eval_mb_size=32,
device=device,
eval_every=1,
plugins=[EarlyStoppingPlugin(args.patience, "test_stream", verbose=True)] << << the default metric_name: str = "Top1_Acc_Stream"
)

# train and test loop
results = []
i = 0
for train_task, test_task in zip(train_stream, test_stream):
print("Current Classes: ", train_task.classes_in_this_experience)
cl_strategy.train(train_task, eval_streams=[test_task])
results.append(cl_strategy.eval(test_stream))
i += 1

`
🐞 **Screenshots**
If applicable, add screenshots to help explain your problem.
Now, l will explain why EarlyStoppingPlugin couldn't train properly for the last experience.
Conclusion: when we training the last experience, EarlyStoppingPlugin will use the fourth experience to determine when the fifth task stops.
image
As shown in the figure above, EarlyStoppingPlugin uses Task003's "Top1_Acc_Stream" to determine Task004's training, not task004 itself, which will result in the Task004 is soon forced to stop early.

In early_stopping.py, the **full_name = names[-1]** of _def **_update_best**_ is used to choose the metric.

I also check the other tasks training situation for the EarlyStoppingPlugin.

- Training experience 0 , EarlyStoppingPlugin using Top1_Acc_Stream/eval_phase/test/Task000 . This is right. becase I will get Top1_Acc_Stream/eval_phase/test/Task000 after **full_name = names[-1]**.
- Training experience 1 , EarlyStoppingPlugin using Top1_Acc_Stream/eval_phase/test/Task001 . This is right. becase I will get Top1_Acc_Stream/eval_phase/test/Task001 after **full_name = names[-1]**.
- Training experience 2 , EarlyStoppingPlugin using Top1_Acc_Stream/eval_phase/test/Task002 . This is right. becase I will get Top1_Acc_Stream/eval_phase/test/Task002 after **full_name = names[-1]**.
- Training experience 3 , EarlyStoppingPlugin using Top1_Acc_Stream/eval_phase/test/Task003 . This is right. becase I will get Top1_Acc_Stream/eval_phase/test/Task003 after **full_name = names[-1]**.
- Training experience 4 , EarlyStoppingPlugin using Top1_Acc_Stream/eval_phase/test/Task003 . This is error. becase I will get Top1_Acc_Stream/eval_phase/test/Task003 after **full_name = names[-1]**.

In summary, the **_full_name = names[-1]_** is not alway choose the right metric.

> I notice that "res = strategy.evaluator.get_last_metrics()" will load the Top1_Acc_Stream/eval_phase/test/Task004 when test the whole steam. Hence, The last item in names is no longer Top1_Acc_Stream/eval_phase/test/Task004 when training experience 4.

Similar, I check the class increamental learning setting as above way, EarlyStoppingPlugin also monitor Top1_Acc_Stream/eval_phase/test/Task000 .

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.