EpistasisLab / EpistasisLab/tpot

Custom scoring function and forkserver

Open
#664 9 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
Jupyter Notebook
Stars
10.1k
Forks
1.6k
PR merge metrics
No merged PRs in 30d

Description

I am using forkserver in order to avoid freezing of TPOT on large dataset. However, i also use a customer scoring function:
```
def get_ranks(arr):
temp = np.argsort(arr)
ranks = np.empty_like(temp).astype('float_')
ranks[temp] = np.arange(len(arr)) * 1.0 / len(arr)
return ranks

def mcc_at_threshold(y, y_pred, threshold):
y_pred_threshold = get_ranks(y_pred) > (1-threshold)
return matthews_corrcoef(y, y_pred_threshold)
cust_scoring_mcc = make_scorer(mcc_at_threshold, needs_threshold=True, threshold=target_churn)
```
when using `scoring = cust_scoring_mcc ` as argument for TPOT, it crashes and i find the following in the traceback:
```
ValueError: 'mcc_at_threshold' is not a valid scoring value. Valid options are ['accuracy', 'adjusted_mutual_info_score', 'adjusted_rand_score', 'average_precision', 'balanced_accuracy', 'completeness_score', 'explained_variance', 'f1', 'f1_macro', 'f1_micro', 'f1_samples', 'f1_weighted', 'fowlkes_mallows_score', 'homogeneity_score', 'mutual_info_score', 'neg_log_loss', 'neg_mean_absolute_error', 'neg_mean_squared_error', 'neg_mean_squared_log_error', 'neg_median_absolute_error', 'normalized_mutual_info_score', 'precision', 'precision_macro', 'precision_micro', 'precision_samples', 'precision_weighted', 'r2', 'recall', 'recall_macro', 'recall_micro', 'recall_samples', 'recall_weighted', 'roc_auc', 'v_measure_score']
```

Any thoughts?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.