EpistasisLab / EpistasisLab/tpot

Make TPOT Robust for Noisy Datasets

Open
#514 13 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
Jupyter Notebook
Stars
10.1k
Forks
1.6k
PR merge metrics
No merged PRs in 30d

Description

I recently raised #513, which had to do with the random_state in TPOT being a (false) contributing factor to the selection of the top pipeline. For the problem referenced there, the problem was not intractable (logistic regression with some preprocessing, a deterministic model) gave 59% classification accuracy, which is actually great.

For highly noisy datasets, I would recommend adding an option to TPOT to randomize the random_state used for the pipelines at each step in the pipeline optimizations (perhaps at each generation or before the scoring function is called). This would make TPOT much more robust for high noise problems, as it would help to prevent good learners from being marginally overtaken by overfit (through the selection of the random_state) learners. For example, a RF with a 'good' random_state beat logistic regression for my problem in #513, but logistic regression was actually learning and generalizing better, and I wouldn't have known that without filtering all of the top genomes in the final population manually by testing them with different random weights.

This is mainly important for being able to use TPOT on financial time series, which are some of the most commonly used datasets to illustrate really hard/highly noisy learning problems. Think it would be great to make TPOT better for this problem class.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.