[Discussion] Today I ran out of memory after running training for 35 hours (spectrogram processing)

I'm working on a large data set (500GB, 100k examples) tuned for splitting drums.

Over the past day and a half I've watched spleeter slowly and steadily take more and more RAM. I think I've loaded up about 128GB of WAVs (assuming the on disk caching is a 1.5 x increase in size compared to the source)... I started to get out of memory errors when I used 72 out of the 84GB of available ram.

When I checked out the dataset.py file, I do think I found that an iterator keeps adding to a variable. Now I'm wondering what is the next best step to handle this. I have some tight constraints on RAM when using a GPU in Google cloud. So I can't just load up with 256GB of ram and call it a day.

I'm confused as to why spleeter does caching to disk but doesn't clear out ram. Can I recover from this cache somehow? What should I expect for RAM sizing?

Obviously my next step is to cut down the training set but eventually I'd like to train it against the full set to see if that produces more accurate results.

Would love some insights as to how I should setup the env. I've burned through about $2k of VM/GPU time.. not something I can keep doing.