EpistasisLab / EpistasisLab/tpot
Model with FeatureUnion with identical FunctionTransformers
- Dominant language
- Jupyter Notebook
- Stars
- 10.1k
- Forks
- 1.6k
- PR merge metrics
- No merged PRs in 30d
Description
My team uses TPOT for model selection as part of ML pipeline to help inform risk opinions for online transactions. We have been using the default classifier configuration provided by TPOT and have for the most part been getting good classifier performance. Our most recent model appeared to have consistent performance against the held-out-set and scoring distribution, but performed poorly serving prediction against out of sample data.
We noticed that this pipeline included a FeatureUnion of twi copy FunctionTransformers. The input to the RF classifier was **double** the width.
```
fu = m37_pipe.steps[0][1]
f0 = fu.transformer_list[0][1]
f1 = fu.transformer_list[1][1]
f0.func == f1.func
```
`True`
All of our prior classifiers have been some form of Tree ensemble with not pre-processing steps.
We have been using TPOT 0.9.0 and the only change prior to our weird model was upgrading Pandas to 0.23.
TPOT configuration used:
```
tpot = TPOTClassifier(generations=5,
population_size=20,
verbosity=0,
n_jobs=1,
config_dict=tpot.config.classifier_config_dict)
```
The fitted TPOT pipeline in question was:
```
[('featureunion', FeatureUnion(n_jobs=1,
transformer_list=[('functiontransformer-1', FunctionTransformer(accept_sparse=False, func=,
inv_kw_args=None, inverse_func=None, kw_args=None,
pass_y='deprecated', validate=True)), ('functiontransformer-2', FunctionTransformer(accept_sparse=False, func=,
inv_kw_args=None, inverse_func=None, kw_args=None,
pass_y='deprecated', validate=True))],
transformer_weights=None)),
('randomforestclassifier', RandomForestClassifier(bootstrap=False, class_weight=None,
criterion='entropy', max_depth=None, max_features=0.2,
max_leaf_nodes=None, min_impurity_decrease=0.0,
min_impurity_split=None, min_samples_leaf=3,
min_samples_split=14, min_weight_fraction_leaf=0.0,
n_estimators=100, n_jobs=1, oob_score=False, random_state=None,
verbose=0, warm_start=False))]
```
I don't really understand why such a preprocessing came about which seems nugator and produced very skewed scoring distribution for out-of-sample predictions.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.