EpistasisLab / EpistasisLab/tpot

Using scoring="recall" results in an error

Open
#1,097 3 comments 0 reactions 0 assignees View on GitHub
unconfirmed
Dominant language
Jupyter Notebook
Stars
10.1k
Forks
1.6k
PR merge metrics
No merged PRs in 30d

Description

I'm trying to use `TPOTClassifier` and it works with `scoring="accuracy"`, but when I try to use `scoring="recall"` the program fails after the second generation with an error.

## Context of the issue

I've been using `TPOTClassifier` successfully for a while but I want to change the scoring metric for the model as I have severely imbalanced data. However, when I change the scoring metric the `TPOTClassifier` fails. I've made sure to run the exact same code with both scoring methods so I know it's an issue with TPOTClassifier instead of my code.

## Process to reproduce the issue

[ordered list the process to finding and recreating the issue, example below]

1. User creates TPOT instance with:
`teapot = TPOTClassifier(max_time_mins=5, scoring="recall")`

2. User calls TPOT `fit()` function with training data:
`teapot.fit(x_train_pp, y_train)`

3. TPOT crashes with this error:

```
RuntimeError Traceback (most recent call last)
in
1 # calculates best model with best hyperparamters
2 teapot = TPOTClassifier(max_time_mins=1, scoring="recall")
----> 3 teapot.fit(x_train_pp, y_train)

~/.local/lib/python3.8/site-packages/tpot/base.py in fit(self, features, target, sample_weight, groups)
754 # raise the exception if it's our last attempt
755 if attempt == (attempts - 1):
--> 756 raise e
757 return self
758

~/.local/lib/python3.8/site-packages/tpot/base.py in fit(self, features, target, sample_weight, groups)
745 self._pbar.close()
746
--> 747 self._update_top_pipeline()
748 self._summary_of_best_pipeline(features, target)
749 # Delete the temporary cache before exiting

~/.local/lib/python3.8/site-packages/tpot/base.py in _update_top_pipeline(self)
805
806 if not self._optimized_pipeline:
--> 807 raise RuntimeError('There was an error in the TPOT optimization '
808 'process. This could be because the data was '
809 'not formatted properly, or because data for '

RuntimeError: There was an error in the TPOT optimization process. This could be because the data was not formatted properly, or because data for a regression problem was provided to the TPOTClassifier object. Please make sure you passed the data to TPOT correctly. If you enabled PyTorch estimators, please check the data requirements in the online documentation: https://epistasislab.github.io/tpot/using/
```

## Expected result

This should have used the recall classifier, which is (True Positives) / (True Positivies + False Negatives).

## Current result\

Currently, using the accuracy metric works perfectly. I can predict using the generated model. Here's my confusion matrix:
`confusion_matrix(y_train_pp, predicted)` outputs:
```
array([[1400, 0],
[ 20, 0]])
```

## Possible fix

If I had to guess I think it might be just a zero-division error when trying to compute the score, resulting in no model being fitted.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.