deezer / deezer/spleeter

[Discussion] Stop training and resume training

Open
#841 6 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
Python
Stars
28.4k
Forks
3.1k
PR merge metrics
No merged PRs in 30d

Description

Hello,
I'm a noob to Spleeter and AI, and am completely going like the Wizard's apprentice here.
I've created a dataset and was excited to see Spleeter starting training. However it was very long and my laptop's fan would sound like an airplane taking off, the computer heated up quite a bit, and after hours of running there was no end in sight.
So, I figured I didn't mind waiting long, but I didn't like my computer heating up. I'm not interested enough to buy a GPU and the non-laptop computer that goes with it to speed up training, so I elected to have a computer on the cloud do the work for me, while I carry on with my life.
However the cloud job has a 1 hour time limit, so I used the caching feature of the online cloud service to cache both my cache folder and model folder between each run: I was hoping every time a new job starts, Spleeter would have the cache and model folders in the state when the last job quit after 1 hour.
It seems to be working, I see a new job pop up every hour and I see a message:
"INFO:tensorflow:Restoring parameters from /myaudiodatabase/myaudiodatabase_model/model.ckpt-0"

Besides that I see a lot of "audio loaded successfully", but only occasionally something about tensorflow, even though I'm using the --verbose option

In case the computation slowed down, I reduced save_checkpoints_steps from 1000 to 100 to get more checkpoints. Is that a good idea ?

Is my approach actually leading to Spleeter eventually finishing, or is my project restarting every hour from the same point ? I'm not 100% sure what I'm doing is actually working. How can I tell when Spleeter is almost finished training ?

Thank you very much.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.