CPU RAM Usage

I'm having issues with the amount of CPU RAM being used by train_ppo.py. Specifically, over the course of training the memory usage steadily increases until there is no memory left, causing an error. This seems odd as I would expect the total memory usage to be roughly constant over the course of training as the model and dataset are both a fixed size. Does anyone have an idea as to why this is the case? Has anyone else experience similar issues, and been able to alleviate them?

Start by reproducing the memory increase while running train_ppo.py and inspect how CPU RAM changes over the course of training. Trace the components that retain memory, then verify that a representative training run maintains roughly constant usage and finishes without exhausting RAM.

Written by the indexing model from the issue text.