EpistasisLab / EpistasisLab/tpot
Lack of reproducibility between TPOTRegressor and .fitted_pipeline_ attribute
- Dominant language
- Jupyter Notebook
- Stars
- 10.1k
- Forks
- 1.6k
- PR merge metrics
- No merged PRs in 30d
Description
It is currently not possible to reproduce the results of regression performed with the TPOTRegressor class with the resulting pipeline.
## Context of the issue
Currently, the accuracy score from the .score() method of a TPOTClassifier instance and the output of sklearn.metrics.accuracy_score on the best pipeline are identical. This is not the case with pipelines from TPOTRegressor instances.
## Process to reproduce the issue
### Classifier (correct/reproducible results)
The following code is used to create a TPOTClassifier, train it on the iris dataset and then return the accuracy
```
from tpot import TPOTClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(iris.data, iris.target,
train_size=0.75, test_size=0.25)
X_train.shape, X_test.shape, y_train.shape, y_test.shape
tpot = TPOTClassifier(verbosity=2, max_time_mins=2)
tpot.fit(X_train, y_train)
print(tpot.score(X_test, y_test))
>>>Optimization Progress: 77%
>>>154/200 [01:49<00:46, 1.00s/pipeline]
>>>2.01 minutes have elapsed. TPOT will close down.
>>>TPOT closed during evaluation in one generation.
>>>WARNING: TPOT may not provide a good pipeline if TPOT is stopped/interrupted in a early generation.
>>>TPOT closed prematurely. Will use the current best pipeline.
>>>Best pipeline: MLPClassifier(input_matrix, alpha=0.01, learning_rate_init=0.001)
>>>1.0
```
When the sklearn.metrics.accuracy_score function is called on the y_test data and the predictions from the best pipeline created by the TPOTClassifier instance, the result is identical:
```
pipeline = tpot.fitted_pipeline_
from sklearn.metrics import accuracy_score
y_pred = pipeline.predict(X_test)
accuracy_score(y_test, y_pred)
>>> 1.0
```
### Regressor (incorrect/nonreproducible results)
With TPOTRegressor, the results are not identical.
```
from tpot import TPOTRegressor
#from sklearn.datasets import load_boston
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_diabetes
X, y = load_diabetes(return_X_y=True)
X = X[:1500]
y = y[:1500]
X_train, X_test, y_train, y_test = train_test_split(X,y,
train_size=0.75, test_size=0.25, random_state=42)
tpot = TPOTRegressor(generations=5, population_size=5, verbosity=2, random_state=42)
tpot.fit(X_train, y_train)
print(-1*tpot.score(X_test, y_test))
>>>Best pipeline: RandomForestRegressor(SelectFromModel(ElasticNetCV(input_matrix, l1_ratio=0.75, tol=0.01), >>>max_features=0.15000000000000002, n_estimators=100, threshold=0.0), bootstrap=True, max_features=0.4, min_samples_leaf=7, >>>min_samples_split=17, n_estimators=100)
>>>2572.133297426151
```
Unfortunately, the results of rerunning the call to .predict with test data for the best pipeline from the TPOTRegressor object, are not identical to this step:
```
from sklearn.metrics import mean_absolute_error
pipeline = tpot.fitted_pipeline_
y_pred = pipeline.predict(X_test)
mean_absolute_error(y_test,y_pred)
>>> 40.08185512072557
```
I would have expected the last line to return 2572.133297426151.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.