EpistasisLab / EpistasisLab/tpot

Custom Scorer (WMAE) fails TPOT optimization process

Open
#1,063 1 comment 0 reactions 0 assignees View on GitHub
question
Dominant language
Jupyter Notebook
Stars
10.1k
Forks
1.6k
PR merge metrics
No merged PRs in 30d

Description

Passing my own scorer that calculates weighted mean absolute error for a regression problem, results in error

## Context of the issue

I followed the instructions from the TPOT documentation page to create my own scorer for a regression problem (I am trying to use tpot for Walmart kaggle competition https://www.kaggle.com/c/walmart-recruiting-store-sales-forecasting). The scorer I created calculates the weighted mean absolute error. I am pasting my code below:

```python
from sklearn.metrics import mean_absolute_error

def mape_score(y_test, y_pred, weights):
mean_absolute_error(y_test, y_pred, sample_weight= weights)

from sklearn.metrics import make_scorer
my_custom_scorer = make_scorer(mape_score, greater_is_better=False)

tpot = TPOTRegressor(generations=5, population_size=50, verbosity=2, random_state=42, n_jobs=-1, scoring=my_custom_scorer)

with joblib.parallel_backend("dask"):
tpot.fit(X_train, y_train)
print(tpot.score(X_test, y_test))
```
weights & y_test & y_pred is type pandas.core.series.Series

I am also using local dask cluster to distribute my workload. Please let me know if it is a user error or something that may need to be looked into.

## Current result
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
/opt/anaconda3/envs/Nelson_Dask/lib/python3.8/site-packages/tpot/base.py in fit(self, features, target, sample_weight, groups)
699 warnings.simplefilter('ignore')
--> 700 self._pop, _ = eaMuPlusLambda(
701 population=self._pop,

/opt/anaconda3/envs/Nelson_Dask/lib/python3.8/site-packages/tpot/gp_deap.py in eaMuPlusLambda(population, toolbox, mu, lambda_, cxpb, mutpb, ngen, pbar, stats, halloffame, verbose, per_generation_function)
235 if per_generation_function is not None:
--> 236 per_generation_function(gen)
237 # Vary the population

/opt/anaconda3/envs/Nelson_Dask/lib/python3.8/site-packages/tpot/base.py in _check_periodic_pipeline(self, gen)
1002 """
-> 1003 self._update_top_pipeline()
1004 if self.periodic_checkpoint_folder is not None:

/opt/anaconda3/envs/Nelson_Dask/lib/python3.8/site-packages/tpot/base.py in _update_top_pipeline(self)
792 if not self._optimized_pipeline:
--> 793 raise RuntimeError('There was an error in the TPOT optimization '
794 'process. This could be because the data was '

RuntimeError: There was an error in the TPOT optimization process. This could be because the data was not formatted properly, or because data for a regression problem was provided to the TPOTClassifier object. Please make sure you passed the data to TPOT correctly.

During handling of the above exception, another exception occurred:

RuntimeError Traceback (most recent call last)
in
1 with joblib.parallel_backend("dask"):
----> 2 tpot.fit(X_train, y_train)
3 print(tpot.score(X_test, y_test))
4 tpot.export('tpot_Nelson_pipeline.py')

/opt/anaconda3/envs/Nelson_Dask/lib/python3.8/site-packages/tpot/base.py in fit(self, features, target, sample_weight, groups)
740 # raise the exception if it's our last attempt
741 if attempt == (attempts - 1):
--> 742 raise e
743 return self
744

/opt/anaconda3/envs/Nelson_Dask/lib/python3.8/site-packages/tpot/base.py in fit(self, features, target, sample_weight, groups)
731 self._pbar.close()
732
--> 733 self._update_top_pipeline()
734 self._summary_of_best_pipeline(features, target)
735 # Delete the temporary cache before exiting

/opt/anaconda3/envs/Nelson_Dask/lib/python3.8/site-packages/tpot/base.py in _update_top_pipeline(self)
791
792 if not self._optimized_pipeline:
--> 793 raise RuntimeError('There was an error in the TPOT optimization '
794 'process. This could be because the data was '
795 'not formatted properly, or because data for '

RuntimeError: There was an error in the TPOT optimization process. This could be because the data was not formatted properly, or because data for a regression problem was provided to the TPOTClassifier object. Please make sure you passed the data to TPOT correctly.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.