EpistasisLab / EpistasisLab/tpot

Some pipelines have huge resource requirements during training?

Open
#1,079 5 comments 1 reaction 0 assignees View on GitHub
need contributor question
Dominant language
Jupyter Notebook
Stars
10.1k
Forks
1.6k
PR merge metrics
No merged PRs in 30d

Description

I've just started using TPOT and am training on a moderately sized dataset of 1000 observations with 5000 predictors (all continuous). During optimization, a few pipelines consume an enormous amount of memory and create huge temporary files (if I've set memory = 'auto') - on the order to 100 GB of memory. (!!)

I've tried to track down the offending type of model but have not been able to do so at this point. I have found that if I use config_dict = 'TPOT light', the problem does not occur. I'm not sure if this is a particular type of model or if it is a particular type of preprocessor (PolynomialFeatures?) or perhaps a combination of those two.

I can use config_dict = 'TPOT light', but I would rather have a larger palette of operators to choose from and merely excise the offending model or preprocessor.

Is there any information regarding what might cause this problem?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.