EpistasisLab / EpistasisLab/tpot
TPOT internal cross validation is not shuffled - can bias results if not accounted for.
- Dominant language
- Jupyter Notebook
- Stars
- 10.1k
- Forks
- 1.6k
- PR merge metrics
- No merged PRs in 30d
Description
TPOTs internal cross validation is not shuffled. Rather, data is split in sequential chunks in the order that it was passed in. (e.g indexes 1-10 is chunk 1, 11-20 is chunk 2, 21-30 is chunk 3, etc.) This could lead to biased results if the data was ordered in a particular way before being given to TPOT, which is common in many cases. TPOT's documentation also does not mention this issue, so users may not know to shuffle their data before passing into TPOT.
TPOT gets its cross validation loop from the check_cv function from sklearn in line [1507 of base.py](https://github.com/EpistasisLab/tpot/blob/master/tpot/base.py#L1507).
This returns either a StratifiedKFold or KFold class ([return statement here](https://github.com/scikit-learn/scikit-learn/blob/32f9deaaf/sklearn/model_selection/_split.py#L2309))
By default these have shuffle set to False. Documentation is for [StratifiedKFold](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedKFold.html) here and [KFold](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.KFold.html) here.
### Possible solutions
Shuffle and random state could be set after defining the cv instance.
`cv.shuffle = True
cv.random_state = self.random_state`
This could also be a user set parameter of TPOTs constructor for clarity.
TPOTs documentation should reference this somehow so that users understand how to format their data before passing it into TPOT.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.