awslabs / awslabs/graphstorm

Add relative increase criterion for early stopping

Open
#1,202 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
452
Forks
76
PR merge metrics
No merged PRs in 30d

Description

Currently early stop has two strategies that only trigger if a validation metric _decreases_ with two criteria:

> There are two strategies: 1) **consecutive_increase**, early stop is triggered if the current validation score is lower than the average of the last early_stop_rounds validation scores and 2) **average_increase**, early stop is triggered if for the last early_stop_rounds consecutive steps, the validation scores are decreasing.

However there is the possibility of a metric becoming "stuck" with very minor increases to its value e.g.

![Image](https://github.com/user-attachments/assets/07fd5202-b69e-43a4-a730-aa49e44a79fd)

To address this case we should add relative increase parameter that will trigger early stop also in the cases where the metric has not improved for more than e.g. 1%. That way we can stop training when a metric plateaus.

Contributor guide

Open the contributing guide

Research direction

Locate the early-stopping implementation that handles the consecutive_increase and average_increase strategies, then read how early_stop_rounds and validation scores are evaluated. Define the relative-increase threshold behavior for a plateau and verify that early stopping triggers when improvement stays below the requested percentage without breaking the existing strategies.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.