GSTT-CSC / GSTT-CSC/project-template

[Feature Request]: resume training from checkpoint

Open
#115 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1
Forks
2
PR merge metrics
No merged PRs in 30d

Description

**Description**
Possibility to resume training from a saved checkpoint.

**Why do you need the new feature?**
Long runs may fail for a variety of reasons - this ensures a run can be resumed with minimal compute time loss.

**Solution suggestion**
Save latest checkpoint more often with nnunet; have a flag in the mlops config (resume_from_checkpoint = ) pointing out to a certain checkpoint in mlflow.

Does this link correctly with the folds? This may require integration with mlflow.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the MLOps configuration and the existing nnUNet checkpoint handling, then review how MLflow stores or identifies checkpoints. Clarify how the resume flag should interact with folds and what integration is required. Done means a failed training run can resume from a selected checkpoint without losing unnecessary compute.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.