EpistasisLab / EpistasisLab/tpot
TPOT underpopulates a class, but manual sklearn does not
- Dominant language
- Jupyter Notebook
- Stars
- 10.1k
- Forks
- 1.6k
- PR merge metrics
- No merged PRs in 30d
Description
## Context of the issue
I have a large and imbalanced binary classification dataset: approx 2,000,000 negative cases and 5,000 positive cases, with 45 features. I have been running manual sklearn pipelines on this dataset without problem for a while. My manual work includes StratifiedKFold cross validation on algorithms such as random forests, gradient boosting, MLP, and more. All of the manual work has been fine.
I recently learned of TPOT (what an awesome idea! huge thanks, devs!) and was excited to give it a try. But on the exact same dataset, I'm getting an error `The least populated class in y has only 1 members, which is less than n_splits=5.` This happens after about 50-60 TPOT iterations. I'm using stratification in train_test_split, and it's just a binary classification. So I'm not sure how a split could end up underpopulated. It's also strange that this same dataset works fine manually with stratification/splitting that (so far as I understand) is identical to what TPOTClassifier uses.
I saw a few other reports of this error both for sklearn and TPOT. But it was always on multilabel classification. So I'm a bit stumped.
## TPOT script
```
#!/usr/bin/env python
from tpot import TPOTClassifier
from sklearn.model_selection import train_test_split, StratifiedKFold
import pandas as pd
import argparse
def get_options():
parser = argparse.ArgumentParser()
parser.add_argument('-f', "--file", required=True, type=str, help="input feature/label file")
parser.add_argument("--ncpu", default=1, type=int, help="number of cpus to use")
args = parser.parse_args()
return args
def load_dataset(filename):
# load the dataset as a pandas dataframe
data = pd.read_csv(filename, sep=",")
data = data.drop(["Chromosome","Start","End","OTseq","guide","Strand"], axis = 'columns')
# split into feature and label elements, where the label is named "autodisco_classifier"
X, y = data.drop(["autodisco", "autodisco_classifier"], axis = 'columns'), data["autodisco_classifier"]
return X, y
args = get_options()
fname = args.file
X, y = load_dataset(fname)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y) # I also tried no stratification here
cv = StratifiedKFold(n_splits=5)
pipeline_optimizer = TPOTClassifier(generations=5, population_size=50, cv=cv, # I also tried just using cv=5
random_state=42, verbosity=2, n_jobs=args.ncpu) # Right now I'm just using one cpu, n_jobs=1
pipeline_optimizer.fit(X_train, y_train)
print(pipeline_optimizer.score(X_test, y_test))
pipeline_optimizer.export('tpot_exported_pipeline.py')
```
## Error
```
Generation 1 - Current best internal CV score: -inf
/home/cornlab/miniconda3/envs/jcorn/lib/python3.10/site-packages/sklearn/model_selection/_split.py:737: UserWarning: The least populated class in y has only 1 members, which is less than n_splits=5.
warnings.warn(
/home/cornlab/miniconda3/envs/jcorn/lib/python3.10/site-packages/sklearn/model_selection/_split.py:737: UserWarning: The least populated class in y has only 1 members, which is less than n_splits=5.
warnings.warn(
/home/cornlab/miniconda3/envs/jcorn/lib/python3.10/site-packages/sklearn/model_selection/_split.py:737: UserWarning: The least populated class in y has only 1 members, which is less than n_splits=5.
warnings.warn(
/home/cornlab/miniconda3/envs/jcorn/lib/python3.10/site-packages/sklearn/model_selection/_split.py:737: UserWarning: The least populated class in y has only 1 members, which is less than n_splits=5.
warnings.warn(
/home/cornlab/miniconda3/envs/jcorn/lib/python3.10/site-packages/sklearn/model_selection/_split.py:737: UserWarning: The least populated class in y has only 1 members, which is less than n_splits=5.
warnings.warn(
/home/cornlab/miniconda3/envs/jcorn/lib/python3.10/site-packages/sklearn/model_selection/_split.py:737: UserWarning: The least populated class in y has only 1 members, which is less than n_splits=5.
warnings.warn(
/home/cornlab/miniconda3/envs/jcorn/lib/python3.10/site-packages/sklearn/model_selection/_split.py:737: UserWarning: The least populated class in y has only 1 members, which is less than n_splits=5.
warnings.warn(
/home/cornlab/miniconda3/envs/jcorn/lib/python3.10/site-packages/sklearn/model_selection/_split.py:737: UserWarning: The least populated class in y has only 1 members, which is less than n_splits=5.
warnings.warn(
/home/cornlab/miniconda3/envs/jcorn/lib/python3.10/site-packages/sklearn/model_selection/_split.py:737: UserWarning: The least populated class in y has only 1 members, which is less than n_splits=5.
warnings.warn(
/home/cornlab/miniconda3/envs/jcorn/lib/python3.10/site-packages/sklearn/model_selection/_split.py:737: UserWarning: The least populated class in y has only 1 members, which is less than n_splits=5.
warnings.warn(
Traceback (most recent call last):
File "/home/cornlab/miniconda3/envs/jcorn/lib/python3.10/site-packages/tpot/base.py", line 817, in fit
self._pop, _ = eaMuPlusLambda(
File "/home/cornlab/miniconda3/envs/jcorn/lib/python3.10/site-packages/tpot/gp_deap.py", line 285, in eaMuPlusLambda
per_generation_function(gen)
File "/home/cornlab/miniconda3/envs/jcorn/lib/python3.10/site-packages/tpot/base.py", line 1183, in _check_periodic_pipeline
self._update_top_pipeline()
File "/home/cornlab/miniconda3/envs/jcorn/lib/python3.10/site-packages/tpot/base.py", line 935, in _update_top_pipeline
raise RuntimeError(
RuntimeError: There was an error in the TPOT optimization process. This could be because the data was not formatted properly, or because data for a regression problem was provided to the TPOTClassifier object. Please make sure you passed the data to TPOT correctly. If you enabled PyTorch estimators, please check the data requirements in the online documentation: https://epistasislab.github.io/tpot/using/
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/mnt/data/jcorn/autodisco/ml/testing/plate2/filtered/learn/../../../../scripts/auto_tpot.py", line 34, in
pipeline_optimizer.fit(X_train, y_train)
File "/home/cornlab/miniconda3/envs/jcorn/lib/python3.10/site-packages/tpot/base.py", line 864, in fit
raise e
File "/home/cornlab/miniconda3/envs/jcorn/lib/python3.10/site-packages/tpot/base.py", line 855, in fit
self._update_top_pipeline()
File "/home/cornlab/miniconda3/envs/jcorn/lib/python3.10/site-packages/tpot/base.py", line 935, in _update_top_pipeline
raise RuntimeError(
RuntimeError: There was an error in the TPOT optimization process. This could be because the data was not formatted properly, or because data for a regression problem was provided to the TPOTClassifier object. Please make sure you passed the data to TPOT correctly. If you enabled PyTorch estimators, please check the data requirements in the online documentation: https://epistasislab.github.io/tpot/using/
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.