EpistasisLab / EpistasisLab/tpot
Using scoring="recall" results in an error
- Dominant language
- Jupyter Notebook
- Stars
- 10.1k
- Forks
- 1.6k
- PR merge metrics
- No merged PRs in 30d
Description
I'm trying to use `TPOTClassifier` and it works with `scoring="accuracy"`, but when I try to use `scoring="recall"` the program fails after the second generation with an error.
## Context of the issue
I've been using `TPOTClassifier` successfully for a while but I want to change the scoring metric for the model as I have severely imbalanced data. However, when I change the scoring metric the `TPOTClassifier` fails. I've made sure to run the exact same code with both scoring methods so I know it's an issue with TPOTClassifier instead of my code.
## Process to reproduce the issue
[ordered list the process to finding and recreating the issue, example below]
1. User creates TPOT instance with:
`teapot = TPOTClassifier(max_time_mins=5, scoring="recall")`
2. User calls TPOT `fit()` function with training data:
`teapot.fit(x_train_pp, y_train)`
3. TPOT crashes with this error:
```
RuntimeError Traceback (most recent call last)
in
1 # calculates best model with best hyperparamters
2 teapot = TPOTClassifier(max_time_mins=1, scoring="recall")
----> 3 teapot.fit(x_train_pp, y_train)
~/.local/lib/python3.8/site-packages/tpot/base.py in fit(self, features, target, sample_weight, groups)
754 # raise the exception if it's our last attempt
755 if attempt == (attempts - 1):
--> 756 raise e
757 return self
758
~/.local/lib/python3.8/site-packages/tpot/base.py in fit(self, features, target, sample_weight, groups)
745 self._pbar.close()
746
--> 747 self._update_top_pipeline()
748 self._summary_of_best_pipeline(features, target)
749 # Delete the temporary cache before exiting
~/.local/lib/python3.8/site-packages/tpot/base.py in _update_top_pipeline(self)
805
806 if not self._optimized_pipeline:
--> 807 raise RuntimeError('There was an error in the TPOT optimization '
808 'process. This could be because the data was '
809 'not formatted properly, or because data for '
RuntimeError: There was an error in the TPOT optimization process. This could be because the data was not formatted properly, or because data for a regression problem was provided to the TPOTClassifier object. Please make sure you passed the data to TPOT correctly. If you enabled PyTorch estimators, please check the data requirements in the online documentation: https://epistasislab.github.io/tpot/using/
```
## Expected result
This should have used the recall classifier, which is (True Positives) / (True Positivies + False Negatives).
## Current result\
Currently, using the accuracy metric works perfectly. I can predict using the generated model. Here's my confusion matrix:
`confusion_matrix(y_train_pp, predicted)` outputs:
```
array([[1400, 0],
[ 20, 0]])
```
## Possible fix
If I had to guess I think it might be just a zero-division error when trying to compute the score, resulting in no model being fitted.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.