EpistasisLab / EpistasisLab/tpot

Having trouble adding classifier metric + question on which metric should be used from existing ones

Open
#1,121 17 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
Jupyter Notebook
Stars
10.1k
Forks
1.6k
PR merge metrics
No merged PRs in 30d

Description

I'm trying to add a couple of binary classification metrics for TPOT. I have a small dataset and a binary classification problem with one class being 40% more than the other. I guess you would call that a moderately unbalanced dataset? I care equally about either of the two classes and I want the metric to be as stringent as possible. From some reading I've done Cohen Kappa and Matthews correlation coefficient are good ones for binary classification and log loss heavily penalizes when the model is confident about something yet wrong.

I'm pretty novice when it comes to programming but I've tried adding Log Loss, for example, using your guide but I get the following error:

`There was an error in the TPOT optimization process. This could be because the data was not formatted properly, or because data for a regression problem was provided to the TPOTClassifier object. Please make sure you passed the data to TPOT correctly. If you enabled PyTorch estimators, please check the data requirements in the online documentation: https://epistasislab.github.io/tpot/using/
`
I've written the following code

```
# Make a custom metric function - LOG Loss
from sklearn.metrics import log_loss
def my_custom_accuracy(y_true, y_pred):
loss = log_loss(y_true,y_pred)
return loss

# Make a custom a scorer from the custom metric function
# Note: greater_is_better=False in make_scorer below would mean that the scoring function should be minimized.
from sklearn.metrics import make_scorer
my_custom_scorer = make_scorer(my_custom_accuracy, greater_is_better=True)
```

In either case, which one do you recommend using from the existing TPOT ones?

Thank you for your help in the matter and hope to hear from you soon. This is the final step of my analysis and I want to get it right... :/

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.