[Discussion] Can't seem to get the GPU working for training; maybe I'm to early in the pipeline to see it?

I've been trying for the past week to get GPU acceleration working and I don't think I'm getting it. I've tried a dedicated VM, the container, version 2.3.0 & 2.2.2.. I've lost count of all the different cuda, tensorflow configurations I've tried.. They all work when I test tensorflow directly for GPU access (I test pyTorch as well) and it seems to have acess.. I thought I got it to work once, I saw GPU memory go up to 20GB (half of the 40GB that was available) but after 6 hours the most I ever saw was the GPU spike up to 15% for 1 sec. Next time I tried the same job on a different config the memory never went above 300MB and GPU usage never went above 0%. When I run nvidia-smi I see the python job listed.

Now I'm not sure if this is because I haven't figured out how to get the GPU setup properly or if there is a very long and costly loading phase (I have about 450k wavs). I see the wavs being loaded in the STDOUT logging, they are getting cached to disk.. But should I be using GPU during this phase or do all the files need to be converted before training really starts?

Can someone please provide a brief explanation of what to expect when the training is working properly.. I can't tell if I'm doing something wrong or not..