ContinualAI / ContinualAI/avalanche
Using validation set for early stopping
- Dominant language
- Python
- Stars
- 2.1k
- Forks
- 321
- PR merge metrics
- No merged PRs in 30d
Description
### Discussed in https://github.com/ContinualAI/avalanche/discussions/900
Originally posted by **sivomke** January 30, 2022
Hi, everyone!
I have created a benchmark with **dataset_benchmark** that contains 3 experiences, and then have added a validation set to this benchmark with **benchmark_with_validation_stream**. I am trying to use Early Stopping with validation set: for each experience, I would like to use loss on the validation set of the corresponding experience. I have indicated **metric_name = 'Loss_Exp'** and **val_stream_name='valid'** , however, it still uses **Loss_Exp/eval_phase/train_stream** of the corresponding experience instead. Could you please tell me what would be the correct way to use validation set for early stopping?
[Here is the script](https://github.com/sivomke/avalanche/blob/main/base-strategy.py)
Thank you!
Contributor guide
Research direction
Start with the linked base-strategy.py script and reproduce the early-stopping setup using metric_name='Loss_Exp' and val_stream_name='valid'. Trace which stream supplies the reported metric; done means early stopping evaluates the corresponding validation experience rather than the training stream.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100