EpistasisLab / EpistasisLab/tpot
Make TPOT Robust for Noisy Datasets
- Dominant language
- Jupyter Notebook
- Stars
- 10.1k
- Forks
- 1.6k
- PR merge metrics
- No merged PRs in 30d
Description
I recently raised #513, which had to do with the random_state in TPOT being a (false) contributing factor to the selection of the top pipeline. For the problem referenced there, the problem was not intractable (logistic regression with some preprocessing, a deterministic model) gave 59% classification accuracy, which is actually great.
For highly noisy datasets, I would recommend adding an option to TPOT to randomize the random_state used for the pipelines at each step in the pipeline optimizations (perhaps at each generation or before the scoring function is called). This would make TPOT much more robust for high noise problems, as it would help to prevent good learners from being marginally overtaken by overfit (through the selection of the random_state) learners. For example, a RF with a 'good' random_state beat logistic regression for my problem in #513, but logistic regression was actually learning and generalizing better, and I wouldn't have known that without filtering all of the top genomes in the final population manually by testing them with different random weights.
This is mainly important for being able to use TPOT on financial time series, which are some of the most commonly used datasets to illustrate really hard/highly noisy learning problems. Think it would be great to make TPOT better for this problem class.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.