GSTT-CSC / GSTT-CSC/project-template
[Feature Request]: resume training from checkpoint
- Dominant language
- Python
- Stars
- 1
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
**Description**
Possibility to resume training from a saved checkpoint.
**Why do you need the new feature?**
Long runs may fail for a variety of reasons - this ensures a run can be resumed with minimal compute time loss.
**Solution suggestion**
Save latest checkpoint more often with nnunet; have a flag in the mlops config (resume_from_checkpoint = ) pointing out to a certain checkpoint in mlflow.
Does this link correctly with the folds? This may require integration with mlflow.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the MLOps configuration and the existing nnUNet checkpoint handling, then review how MLflow stores or identifies checkpoints. Clarify how the resume flag should interact with folds and what integration is required. Done means a failed training run can resume from a selected checkpoint without losing unnecessary compute.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100