EpistasisLab / EpistasisLab/tpot

Lack of reproducibility between TPOTRegressor and .fitted_pipeline_ attribute

Open
#1,305 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
10.1k
Forks
1.6k
PR merge metrics
No merged PRs in 30d

Description

It is currently not possible to reproduce the results of regression performed with the TPOTRegressor class with the resulting pipeline.

## Context of the issue
Currently, the accuracy score from the .score() method of a TPOTClassifier instance and the output of sklearn.metrics.accuracy_score on the best pipeline are identical. This is not the case with pipelines from TPOTRegressor instances.

## Process to reproduce the issue

### Classifier (correct/reproducible results)

The following code is used to create a TPOTClassifier, train it on the iris dataset and then return the accuracy

```
from tpot import TPOTClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(iris.data, iris.target,
train_size=0.75, test_size=0.25)
X_train.shape, X_test.shape, y_train.shape, y_test.shape
tpot = TPOTClassifier(verbosity=2, max_time_mins=2)
tpot.fit(X_train, y_train)
print(tpot.score(X_test, y_test))

>>>Optimization Progress: 77%
>>>154/200 [01:49<00:46, 1.00s/pipeline]

>>>2.01 minutes have elapsed. TPOT will close down.
>>>TPOT closed during evaluation in one generation.
>>>WARNING: TPOT may not provide a good pipeline if TPOT is stopped/interrupted in a early generation.

>>>TPOT closed prematurely. Will use the current best pipeline.

>>>Best pipeline: MLPClassifier(input_matrix, alpha=0.01, learning_rate_init=0.001)
>>>1.0

```

When the sklearn.metrics.accuracy_score function is called on the y_test data and the predictions from the best pipeline created by the TPOTClassifier instance, the result is identical:

```
pipeline = tpot.fitted_pipeline_
from sklearn.metrics import accuracy_score
y_pred = pipeline.predict(X_test)
accuracy_score(y_test, y_pred)
>>> 1.0
```
### Regressor (incorrect/nonreproducible results)
With TPOTRegressor, the results are not identical.
```
from tpot import TPOTRegressor
#from sklearn.datasets import load_boston
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_diabetes

X, y = load_diabetes(return_X_y=True)

X = X[:1500]
y = y[:1500]

X_train, X_test, y_train, y_test = train_test_split(X,y,
train_size=0.75, test_size=0.25, random_state=42)

tpot = TPOTRegressor(generations=5, population_size=5, verbosity=2, random_state=42)
tpot.fit(X_train, y_train)
print(-1*tpot.score(X_test, y_test))
>>>Best pipeline: RandomForestRegressor(SelectFromModel(ElasticNetCV(input_matrix, l1_ratio=0.75, tol=0.01), >>>max_features=0.15000000000000002, n_estimators=100, threshold=0.0), bootstrap=True, max_features=0.4, min_samples_leaf=7, >>>min_samples_split=17, n_estimators=100)
>>>2572.133297426151
```
Unfortunately, the results of rerunning the call to .predict with test data for the best pipeline from the TPOTRegressor object, are not identical to this step:

```
from sklearn.metrics import mean_absolute_error
pipeline = tpot.fitted_pipeline_
y_pred = pipeline.predict(X_test)
mean_absolute_error(y_test,y_pred)
>>> 40.08185512072557
```
I would have expected the last line to return 2572.133297426151.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.