EpistasisLab / EpistasisLab/tpot
Some pipelines have huge resource requirements during training?
- Dominant language
- Jupyter Notebook
- Stars
- 10.1k
- Forks
- 1.6k
- PR merge metrics
- No merged PRs in 30d
Description
I've just started using TPOT and am training on a moderately sized dataset of 1000 observations with 5000 predictors (all continuous). During optimization, a few pipelines consume an enormous amount of memory and create huge temporary files (if I've set memory = 'auto') - on the order to 100 GB of memory. (!!)
I've tried to track down the offending type of model but have not been able to do so at this point. I have found that if I use config_dict = 'TPOT light', the problem does not occur. I'm not sure if this is a particular type of model or if it is a particular type of preprocessor (PolynomialFeatures?) or perhaps a combination of those two.
I can use config_dict = 'TPOT light', but I would rather have a larger palette of operators to choose from and merely excise the offending model or preprocessor.
Is there any information regarding what might cause this problem?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.