EpistasisLab / EpistasisLab/tpot

Smart seeding of TPOT populations?

Open
#59 4 comments 0 reactions 0 assignees View on GitHub
enhancement need contributor
Dominant language
Jupyter Notebook
Stars
10.1k
Forks
1.6k
PR merge metrics
No merged PRs in 30d

Description

Sorry if the text below sounds like rambling -- I was using this issue to brainstorm.

I've been thinking about possible ways to make TPOT perform better right out of the box, without having to run it for several generations to finally discover the better pipelines. One of the ideas I've had is to seed the TPOT population with a smarter group of solutions.

For example, we know that a TPOT pipeline will need at least one model, so we can seed it with each of the 6 current models over a small range of parameters:
- decision tree: all combinations of
- max_features = [0 (--> auto), 1 (--> None)]
- max_depth: [0 (--> None), 1, 5, 10, 20, 50]
- = 12 total combinations
- random forest: all combinations of
- n_estimators = [100, 500]
- max_features = [0 (--> auto), 1]
- = 4 total combinations
- logistic regression:
- C = [0.01, 0.1, 0.5, 1.0, 10.0, 50.0, 100.0]
- = 7 total combinations
- svc:
- C = [0.01, 0.1, 0.5, 1.0, 10.0, 50.0, 100.0]
- = 7 total combinations
- knnc:
- n_neighbors = [2, 5, 10, 20, 50]
- = 5 total combinations
- gradient boosting: all combinations of
- learning_rate: [0.01, 0.1, 0.5, 1.0]
- n_estimators: [100, 500]
- max_depth: [0 (--> None), 5, 10]
- = 24 total combinations

That gives us 59 "classifier-only" TPOT pipelines to start with.

We also have 4 feature selectors:
- RFE: all combinations of
- num_features = [1, 5, 10, 50]
- step = [0.1, 0.25, 0.5]
- = 12 total combinations
- select percentile:
- percentile = [1, 5, 10, 25, 50, 75]
- = 6 total combinations
- select k best:
- k = [1, 2, 5, 10, 20, 50]
- = 6 total combinations
- variance threshold:
- threshold = [0.1, 0.2, 0.3, 0.4, 0.5]
- = 5 total combinations

And 4 feature preprocessors:
- standard scaler (no parameters)
- = 1 total combinations
- robust scaler (no parameters)
- = 1 total combinations
- polynomial features (no parameters)
- = 1 total combinations
- PCA:
- n_components = [1, 2, 4, 10, 20]
- = 5 total combinations

Thus, if we wanted to provide at least one feature preprocessor _or_ selector in the pipeline before passing the data to the model, that would result in:

feature selection combinations = 12 \* 59 + 6 \* 59 + 6 \* 59 + 5 \* 59 = 1,711

feature preprocessor combinations = 5 \* 59 + 1 \* 59 + 1 \* 59 + 1 \* 59 = 472

Giving us a total = 59 + 1,711 + 472 = 2,242 pipeline combinations to start out with.

We'd evaluate all 2,242 of these pipelines then use the top 100 to seed the TPOT population. From there, the GP algorithm is allowed to tinker with the pipeline, fine-tune the parameters, and possibly discover better combinations of pipeline operators.

That's obviously a lot of pipelines to try out at the beginning -- about 23 generations worth of pipelines, which will be quite slow on any decently sized data set. It may be necessary to cut down on the parameters that we try out at the beginning.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.