EpistasisLab / EpistasisLab/tpot

Can`t reproduce the score after refitting TPOT regressor

Open
#1,206 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
10.1k
Forks
1.6k
PR merge metrics
No merged PRs in 30d

Description

Hi
The exported pipeline of TPOT stating that the 'Average CV score on the training set was: -128.90187963562252' (neg_MAE)
However, refitting the pipeline with the same exact training set yields way too smaller MAE around (35).
Moreover predicting unseen test set would yield an MAE around (140 ) which is in line with what the exported pipeline stating.
I am a bit confused and wondering how to reproduce the error score on the training set.
The pipeline seems to be overfitting right??
here is a piece of code used to get the pipeline


cv = RepeatedKFold(n_splits=4, n_repeats=1, random_state=1)
model = TPOTRegressor(generations=10, population_size=25, offspring_size=None, mutation_rate=0.9,
crossover_rate=0.1, scoring='neg_mean_absolute_error', cv=cv, subsample=0.75,
n_jobs=-1, max_time_mins=None, max_eval_time_mins=5,random_state=42,
config_dict=None, template=None, warm_start=False, memory=None, use_dask=False,
periodic_checkpoint_folder=None, early_stop=3, verbosity=2,
disable_update_check=False, log_file=None
)
model.fit(train_df[x], train_df[y])

The Exported model

# Average CV score on the training set was: -128.90187963562252
exported_pipeline = make_pipeline(
StackingEstimator(estimator=LassoLarsCV(normalize=True)),
StackingEstimator(estimator=ExtraTreesRegressor(bootstrap=True, max_features=0.4, min_samples_leaf=1,
min_samples_split=7, n_estimators=100)),
PolynomialFeatures(degree=2, include_bias=False, interaction_only=False),
ExtraTreesRegressor(bootstrap=True, max_features=0.15000000000000002, min_samples_leaf=9, min_samples_split=7,
n_estimators=100))
# Fix random state for all the steps in exported pipeline
set_param_recursive(exported_pipeline.steps, 'random_state', 42)
exported_pipeline.fit(training_features, training_target)
results = exported_pipeline.predict(testing_features)

Thanks in advance

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.