RVC-Project / RVC-Project/Retrieval-based-Voice-Conversion-WebUI
[Feature Request] Save the last n checkpoint
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 38.4k
- Forks
- 5.3k
- PR merge metrics
- No merged PRs in 30d
Description
While saving only the latest '.ckpt' file to save disk space is a great idea, there are several situations where something unexpected happens like some error in Colab that is corrupting latest D/G_2333333.pth file (not saving it fully), or saving only D while not having time to save G checkpoint (or vise versa) because it was interrupted.
I trained a few models using Colab and a lot of them had corrupted D or G checkpoint because some error happened while it was saving, so i lost hours of trained data and need to start from scratch which is very annoying.
My idea is to have some option inside UI like "Save last n checkpoints". As an example, if you set this option to 10 and the 'Save frequency (save_every_epoch)' set to 10, it will save 10, 20, 30 ... 100 epoch D and G checkpoints, then it will delete oldest checkpoints (D_10.pth and G_10.pth in this example), and save latest one (save 110, then delete 20 epoch checkpoints and save 120 and continue). That way we will have some backups and can continue to train model from a older checkpoint if something bad happened with latest checkpoint.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the training UI option and the save_every_epoch checkpoint flow, including where D_.pth and G_.pth files are written. Check how interrupted saves are handled and define the retention behavior for paired checkpoints. Done means an option retains the requested number of recent checkpoints and removes older ones without breaking resume training.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100