EpistasisLab / EpistasisLab/tpot
ValueError: y_true and y_pred contain different number of classes 2, 3.
- Dominant language
- Jupyter Notebook
- Stars
- 10.1k
- Forks
- 1.6k
- PR merge metrics
- No merged PRs in 30d
Description
A `TPOT.fit` call may fail when there are outlier minority classes (with certain metrics).
## Context of the issue
When running the benchmark we encountered this issue sometimes, for instance with evaluations on [`wine-quality-white`](https://www.openml.org/d/40498): `python runbenchmark.py TPOT openml/t/359974 1h8c -f 6`. Because of TPOT internals, the small minority classes may cause an error when optimizing towards log loss. I reduced the issue to a minimal example:
```python
from tpot import TPOTClassifier
import numpy as np
x, y = np.random.random((151, 4)), np.asarray([0] * 75 + [1] * 75 + [2])
t = TPOTClassifier(max_time_mins=1, scoring="neg_log_loss")
t.fit(x,y)
t.predict(x)
```
## Expected result
I expect a pipeline to be fit regardless, and be able to produce predictions for every class (even if that means with a probability of zero and receiving a warning about it).
## Current result
Running the MWE:
```
(venv) root@69ae78289b07:/bench# python mwe.py
/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/sklearn/model_selection/_split.py:668: UserWarning: The least populated class in y has only 1 members, which is less than n_splits=5.
% (min_groups, self.n_splits)), UserWarning)
/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/xgboost/sklearn.py:1146: UserWarning: The use of label encoder in XGBClassifier is deprecated and will be removed in a future release. To remove this warning, do the following: 1) Pass option use_label_encoder=False when constructing XGBClassifier object; and 2) Encode your labels (y) as integers starting with 0, i.e. 0, 1, 2, ..., [num_class - 1].
warnings.warn(label_encoder_deprecation_msg, UserWarning)
/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/sklearn/model_selection/_split.py:668: UserWarning: The least populated class in y has only 1 members, which is less than n_splits=5.
% (min_groups, self.n_splits)), UserWarning)
/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/xgboost/sklearn.py:1146: UserWarning: The use of label encoder in XGBClassifier is deprecated and will be removed in a future release. To remove this warning, do the following: 1) Pass option use_label_encoder=False when constructing XGBClassifier object; and 2) Encode your labels (y) as integers starting with 0, i.e. 0, 1, 2, ..., [num_class - 1].
warnings.warn(label_encoder_deprecation_msg, UserWarning)
/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/sklearn/model_selection/_split.py:668: UserWarning: The least populated class in y has only 1 members, which is less than n_splits=5.
% (min_groups, self.n_splits)), UserWarning)
/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/xgboost/sklearn.py:1146: UserWarning: The use of label encoder in XGBClassifier is deprecated and will be removed in a future release. To remove this warning, do the following: 1) Pass option use_label_encoder=False when constructing XGBClassifier object; and 2) Encode your labels (y) as integers starting with 0, i.e. 0, 1, 2, ..., [num_class - 1].
warnings.warn(label_encoder_deprecation_msg, UserWarning)
/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/sklearn/model_selection/_split.py:668: UserWarning: The least populated class in y has only 1 members, which is less than n_splits=5.
% (min_groups, self.n_splits)), UserWarning)
/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/xgboost/sklearn.py:1146: UserWarning: The use of label encoder in XGBClassifier is deprecated and will be removed in a future release. To remove this warning, do the following: 1) Pass option use_label_encoder=False when constructing XGBClassifier object; and 2) Encode your labels (y) as integers starting with 0, i.e. 0, 1, 2, ..., [num_class - 1].
warnings.warn(label_encoder_deprecation_msg, UserWarning)
/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/sklearn/model_selection/_split.py:668: UserWarning: The least populated class in y has only 1 members, which is less than n_splits=5.
% (min_groups, self.n_splits)), UserWarning)
/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/xgboost/sklearn.py:1146: UserWarning: The use of label encoder in XGBClassifier is deprecated and will be removed in a future release. To remove this warning, do the following: 1) Pass option use_label_encoder=False when constructing XGBClassifier object; and 2) Encode your labels (y) as integers starting with 0, i.e. 0, 1, 2, ..., [num_class - 1].
warnings.warn(label_encoder_deprecation_msg, UserWarning)
/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/sklearn/model_selection/_split.py:668: UserWarning: The least populated class in y has only 1 members, which is less than n_splits=5.
% (min_groups, self.n_splits)), UserWarning)
/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/xgboost/sklearn.py:1146: UserWarning: The use of label encoder in XGBClassifier is deprecated and will be removed in a future release. To remove this warning, do the following: 1) Pass option use_label_encoder=False when constructing XGBClassifier object; and 2) Encode your labels (y) as integers starting with 0, i.e. 0, 1, 2, ..., [num_class - 1].
warnings.warn(label_encoder_deprecation_msg, UserWarning)
/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/sklearn/model_selection/_split.py:668: UserWarning: The least populated class in y has only 1 members, which is less than n_splits=5.
% (min_groups, self.n_splits)), UserWarning)
/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/xgboost/sklearn.py:1146: UserWarning: The use of label encoder in XGBClassifier is deprecated and will be removed in a future release. To remove this warning, do the following: 1) Pass option use_label_encoder=False when constructing XGBClassifier object; and 2) Encode your labels (y) as integers starting with 0, i.e. 0, 1, 2, ..., [num_class - 1].
warnings.warn(label_encoder_deprecation_msg, UserWarning)
/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/sklearn/model_selection/_split.py:668: UserWarning: The least populated class in y has only 1 members, which is less than n_splits=5.
% (min_groups, self.n_splits)), UserWarning)
/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/xgboost/sklearn.py:1146: UserWarning: The use of label encoder in XGBClassifier is deprecated and will be removed in a future release. To remove this warning, do the following: 1) Pass option use_label_encoder=False when constructing XGBClassifier object; and 2) Encode your labels (y) as integers starting with 0, i.e. 0, 1, 2, ..., [num_class - 1].
warnings.warn(label_encoder_deprecation_msg, UserWarning)
/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/sklearn/model_selection/_split.py:668: UserWarning: The least populated class in y has only 1 members, which is less than n_splits=5.
% (min_groups, self.n_splits)), UserWarning)
/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/xgboost/sklearn.py:1146: UserWarning: The use of label encoder in XGBClassifier is deprecated and will be removed in a future release. To remove this warning, do the following: 1) Pass option use_label_encoder=False when constructing XGBClassifier object; and 2) Encode your labels (y) as integers starting with 0, i.e. 0, 1, 2, ..., [num_class - 1].
warnings.warn(label_encoder_deprecation_msg, UserWarning)
/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/sklearn/model_selection/_split.py:668: UserWarning: The least populated class in y has only 1 members, which is less than n_splits=5.
% (min_groups, self.n_splits)), UserWarning)
/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/xgboost/sklearn.py:1146: UserWarning: The use of label encoder in XGBClassifier is deprecated and will be removed in a future release. To remove this warning, do the following: 1) Pass option use_label_encoder=False when constructing XGBClassifier object; and 2) Encode your labels (y) as integers starting with 0, i.e. 0, 1, 2, ..., [num_class - 1].
warnings.warn(label_encoder_deprecation_msg, UserWarning)
Traceback (most recent call last):
File "/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/tpot/base.py", line 828, in fit
log_file=self.log_file_,
File "/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/tpot/gp_deap.py", line 281, in eaMuPlusLambda
per_generation_function(gen)
File "/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/tpot/base.py", line 1176, in _check_periodic_pipeline
self._update_top_pipeline()
File "/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/tpot/base.py", line 931, in _update_top_pipeline
error_score="raise",
File "/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/sklearn/utils/validation.py", line 63, in inner_f
return f(*args, **kwargs)
File "/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/sklearn/model_selection/_validation.py", line 450, in cross_val_score
error_score=error_score)
File "/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/sklearn/utils/validation.py", line 63, in inner_f
return f(*args, **kwargs)
File "/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/sklearn/model_selection/_validation.py", line 256, in cross_validate
for train, test in cv.split(X, y, groups))
File "/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/joblib/parallel.py", line 1041, in __call__
if self.dispatch_one_batch(iterator):
File "/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/joblib/parallel.py", line 859, in dispatch_one_batch
self._dispatch(tasks)
File "/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/joblib/parallel.py", line 777, in _dispatch
job = self._backend.apply_async(batch, callback=cb)
File "/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/joblib/_parallel_backends.py", line 208, in apply_async
result = ImmediateResult(func)
File "/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/joblib/_parallel_backends.py", line 572, in __init__
self.results = batch()
File "/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/joblib/parallel.py", line 263, in __call__
for func, args, kwargs in self.items]
File "/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/joblib/parallel.py", line 263, in
for func, args, kwargs in self.items]
File "/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/sklearn/utils/fixes.py", line 222, in __call__
return self.function(*args, **kwargs)
File "/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/sklearn/model_selection/_validation.py", line 625, in _fit_and_score
test_scores = _score(estimator, X_test, y_test, scorer, error_score)
File "/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/sklearn/model_selection/_validation.py", line 687, in _score
scores = scorer(estimator, X_test, y_test)
File "/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/sklearn/metrics/_scorer.py", line 88, in __call__
*args, **kwargs)
File "/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/sklearn/metrics/_scorer.py", line 288, in _score
return self._sign * self._score_func(y, y_pred, **self._kwargs)
File "/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/sklearn/utils/validation.py", line 63, in inner_f
return f(*args, **kwargs)
File "/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/sklearn/metrics/_classification.py", line 2275, in log_loss
lb.classes_))
ValueError: y_true and y_pred contain different number of classes 2, 3. Please provide the true labels explicitly through the labels argument. Classes found in y_true: [0 1]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "mwe.py", line 6, in
t.fit(x,y)
File "/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/tpot/base.py", line 863, in fit
raise e
File "/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/tpot/base.py", line 854, in fit
self._update_top_pipeline()
File "/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/tpot/base.py", line 931, in _update_top_pipeline
error_score="raise",
File "/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/sklearn/utils/validation.py", line 63, in inner_f
return f(*args, **kwargs)
File "/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/sklearn/model_selection/_validation.py", line 450, in cross_val_score
error_score=error_score)
File "/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/sklearn/utils/validation.py", line 63, in inner_f
return f(*args, **kwargs)
File "/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/sklearn/model_selection/_validation.py", line 256, in cross_validate
for train, test in cv.split(X, y, groups))
File "/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/joblib/parallel.py", line 1041, in __call__
if self.dispatch_one_batch(iterator):
File "/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/joblib/parallel.py", line 859, in dispatch_one_batch
self._dispatch(tasks)
File "/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/joblib/parallel.py", line 777, in _dispatch
job = self._backend.apply_async(batch, callback=cb)
File "/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/joblib/_parallel_backends.py", line 208, in apply_async
result = ImmediateResult(func)
File "/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/joblib/_parallel_backends.py", line 572, in __init__
self.results = batch()
File "/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/joblib/parallel.py", line 263, in __call__
for func, args, kwargs in self.items]
File "/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/joblib/parallel.py", line 263, in
for func, args, kwargs in self.items]
File "/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/sklearn/utils/fixes.py", line 222, in __call__
return self.function(*args, **kwargs)
File "/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/sklearn/model_selection/_validation.py", line 625, in _fit_and_score
test_scores = _score(estimator, X_test, y_test, scorer, error_score)
File "/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/sklearn/model_selection/_validation.py", line 687, in _score
scores = scorer(estimator, X_test, y_test)
File "/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/sklearn/metrics/_scorer.py", line 88, in __call__
*args, **kwargs)
File "/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/sklearn/metrics/_scorer.py", line 288, in _score
return self._sign * self._score_func(y, y_pred, **self._kwargs)
File "/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/sklearn/utils/validation.py", line 63, in inner_f
return f(*args, **kwargs)
File "/bench/frameworks/TPOT/venv/lib/python3.7/site-packages/sklearn/metrics/_classification.py", line 2275, in log_loss
lb.classes_))
ValueError: y_true and y_pred contain different number of classes 2, 3. Please provide the true labels explicitly through the labels argument. Classes found in y_true: [0 1]
```
## Possible fix
Depends on the level you want to fix it on, options include:
- training a separate model for outlier detection of the super-small minority classes and fitting a pipeline for the remainder of the data
- ignoring the data for the problematic classes altogether while emitting an explicit warning about it (which is different from the current scenario where there are only `scikit-learn` warnings, and also lead to the error)
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.