RVC-Project / RVC-Project/Retrieval-based-Voice-Conversion-WebUI

[Feature Request] Save the last n checkpoint

Open
#688 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

following up todo list
Dominant language
Python
Stars
38.4k
Forks
5.3k
PR merge metrics
No merged PRs in 30d

Description

While saving only the latest '.ckpt' file to save disk space is a great idea, there are several situations where something unexpected happens like some error in Colab that is corrupting latest D/G_2333333.pth file (not saving it fully), or saving only D while not having time to save G checkpoint (or vise versa) because it was interrupted.

I trained a few models using Colab and a lot of them had corrupted D or G checkpoint because some error happened while it was saving, so i lost hours of trained data and need to start from scratch which is very annoying.

My idea is to have some option inside UI like "Save last n checkpoints". As an example, if you set this option to 10 and the 'Save frequency (save_every_epoch)' set to 10, it will save 10, 20, 30 ... 100 epoch D and G checkpoints, then it will delete oldest checkpoints (D_10.pth and G_10.pth in this example), and save latest one (save 110, then delete 20 epoch checkpoints and save 120 and continue). That way we will have some backups and can continue to train model from a older checkpoint if something bad happened with latest checkpoint.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the training UI option and the save_every_epoch checkpoint flow, including where D_.pth and G_.pth files are written. Check how interrupted saves are handled and define the retention behavior for paired checkpoints. Done means an option retains the requested number of recent checkpoints and removes older ones without breaking resume training.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.