huggingface / huggingface/setfit

EarlyStoppingCallback early_stopping_patience_counter and Trainer.state not reset between hyperparameter search trials

Open
#502 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
2.8k
Forks
267
Avg merge
36m
Merged PRs (30d)
5

Description

If using the EarlyStoppingCallback from the transformers package, and running the `Trainer.hyperparameter_search` method, `early_stopping_patience_counter` on the callback instance and `Trainer.state` are not reset between subsequent trials.

This means that when running the second and subsequent trials, the `early_stopping_patience_counter` is greater than the `early_stopping_patience`. This causes `Trainer.control.should_training_stop` to be set to `True` the first time `on_evaluate` is run on the callbacks. The trial is then terminated.

To resolve this issue, I suggest that when calling `Trainer.train(trial=...)` and passing in a trial, `Trainer.callback_hander` should be re-initialised and `Trainer.state` initialised to `TrainerState()`.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.