Prophet search runs out of memory when running on my 16GB laptop
- Dominant language
- Python
- Stars
- 20.4k
- Forks
- 4.6k
- Avg merge
- 19h 52m
- Merged PRs (30d)
- 1
Description
I've got a problem with my python script that takes this enourmous amount of my RAM memory. I runned the script on a dell laptop with 16GB of RAM memory, in an ubuntu enviroment.
The problem is that, after some time the proccess of python occupies 8-9-10 GB of my RAM memory than it crashes and in the terminal I can see that the process has been killed.
Explaining the script:
The script that I've created is meant for training a prophet model for aproximatelly 1500 products. For each product the script creates dinamically a folder with the product name and inside of that folder saves 10 files (which help me to see how the model performed on that data that was given).
Also it takes to some preprocess steps where I compute the df that are about to be given to the prophet model. Here I add lagged regressors based on the previous values of the df given, replace the outliers values, and padding with 0 for the missing dates in the df.
After that, I train the prophet model (using the default parameters, so no search for the best hyperparameters) and save various files in the product folder (files ----> MAPEs.txt, MSEs.txt, plotcomponents.png, a plot between actuals vs predicted etc..) in total 10 files for each product. I'm 100% sure that it opens and closes corecttly each file.
I've made 2 scripts --- 1 called general.py (Ik that's not the best name but i couldn't find something more intuitive at that time) and 2. utils.py (here's where all the preprocess functions and training functions are stored). I only run general.py bcs from utils I need only the functions to call them in general.py. After about 70-80 products it crashes killing the python process.
What I've tried also was deleting using del from python each variable at the end of each function from utils.py, leaving only the ones that I want to return them. After that I also used garbage collector with gc.collect() bcs I've read that otherwise python will not instantly restore the memory to the CPU. However, this didn't seem very helpfull at all.
Any suggestions that I can make it run on all products without occupying this large amount of memory?
Contributor guide
Assessment
This issue has not been assessed yet.