[Discussion] Stop training and resume training
- Dominant language
- Python
- Stars
- 28.4k
- Forks
- 3.1k
- PR merge metrics
- No merged PRs in 30d
Description
Hello,
I'm a noob to Spleeter and AI, and am completely going like the Wizard's apprentice here.
I've created a dataset and was excited to see Spleeter starting training. However it was very long and my laptop's fan would sound like an airplane taking off, the computer heated up quite a bit, and after hours of running there was no end in sight.
So, I figured I didn't mind waiting long, but I didn't like my computer heating up. I'm not interested enough to buy a GPU and the non-laptop computer that goes with it to speed up training, so I elected to have a computer on the cloud do the work for me, while I carry on with my life.
However the cloud job has a 1 hour time limit, so I used the caching feature of the online cloud service to cache both my cache folder and model folder between each run: I was hoping every time a new job starts, Spleeter would have the cache and model folders in the state when the last job quit after 1 hour.
It seems to be working, I see a new job pop up every hour and I see a message:
"INFO:tensorflow:Restoring parameters from /myaudiodatabase/myaudiodatabase_model/model.ckpt-0"
Besides that I see a lot of "audio loaded successfully", but only occasionally something about tensorflow, even though I'm using the --verbose option
In case the computation slowed down, I reduced save_checkpoints_steps from 1000 to 100 to get more checkpoints. Is that a good idea ?
Is my approach actually leading to Spleeter eventually finishing, or is my project restarting every hour from the same point ? I'm not 100% sure what I'm doing is actually working. How can I tell when Spleeter is almost finished training ?
Thank you very much.
Contributor guide
Assessment
This issue has not been assessed yet.