Add relative increase criterion for early stopping
- Dominant language
- Python
- Stars
- 452
- Forks
- 76
- PR merge metrics
- No merged PRs in 30d
Description
Currently early stop has two strategies that only trigger if a validation metric _decreases_ with two criteria:
> There are two strategies: 1) **consecutive_increase**, early stop is triggered if the current validation score is lower than the average of the last early_stop_rounds validation scores and 2) **average_increase**, early stop is triggered if for the last early_stop_rounds consecutive steps, the validation scores are decreasing.
However there is the possibility of a metric becoming "stuck" with very minor increases to its value e.g.

To address this case we should add relative increase parameter that will trigger early stop also in the cases where the metric has not improved for more than e.g. 1%. That way we can stop training when a metric plateaus.
Contributor guide
Research direction
Locate the early-stopping implementation that handles the consecutive_increase and average_increase strategies, then read how early_stop_rounds and validation scores are evaluated. Define the relative-increase threshold behavior for a plateau and verify that early stopping triggers when improvement stays below the requested percentage without breaking the existing strategies.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100