EpistasisLab / EpistasisLab/tpot

TPOT internal cross validation is not shuffled - can bias results if not accounted for.

Open
#1,249 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
10.1k
Forks
1.6k
PR merge metrics
No merged PRs in 30d

Description

TPOTs internal cross validation is not shuffled. Rather, data is split in sequential chunks in the order that it was passed in. (e.g indexes 1-10 is chunk 1, 11-20 is chunk 2, 21-30 is chunk 3, etc.) This could lead to biased results if the data was ordered in a particular way before being given to TPOT, which is common in many cases. TPOT's documentation also does not mention this issue, so users may not know to shuffle their data before passing into TPOT.

TPOT gets its cross validation loop from the check_cv function from sklearn in line [1507 of base.py](https://github.com/EpistasisLab/tpot/blob/master/tpot/base.py#L1507).

This returns either a StratifiedKFold or KFold class ([return statement here](https://github.com/scikit-learn/scikit-learn/blob/32f9deaaf/sklearn/model_selection/_split.py#L2309))

By default these have shuffle set to False. Documentation is for [StratifiedKFold](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedKFold.html) here and [KFold](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.KFold.html) here.

### Possible solutions

Shuffle and random state could be set after defining the cv instance.

`cv.shuffle = True
cv.random_state = self.random_state`

This could also be a user set parameter of TPOTs constructor for clarity.

TPOTs documentation should reference this somehow so that users understand how to format their data before passing it into TPOT.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.