GPU memory issues (leak?)

I am running a loop where I initialize a new model in each loop and train it. I am using NVIDIA Apex for mixed precision training. My current issue is that there seems to be some unwanted memory allocations across different steps in the loop. The GPU memory accumulates and after a few steps in the loop CUDA memory runs out.

I have debugged everything, have monitored memory, and have deleted every single thing possible. Only after removing apex the memory allocation seems to be consistent. I am doing nothing else than adding the three lines of code from the tutorial for initializing and backward passing.

Any ideas?

Start by reproducing the reported loop that initializes and trains a new model on each iteration with NVIDIA Apex mixed precision, using the three tutorial lines for initialization and backward passing. Monitor GPU allocations across iterations and compare the behavior with Apex enabled and removed; done means memory no longer accumulates until CUDA runs out.