huggingface / huggingface/setfit

Support metrics using prediction_scores in `SetFitTrainer.evaluate`

Open
#241 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
2.8k
Forks
267
Avg merge
36m
Merged PRs (30d)
5

Description

Can I make a request for `SetFitTrainer.evaluate` to allow the calculation of metrics that take prediction scores as input, i.e., the output of `self.model.predict_proba`? The relevant line is this one: https://github.com/huggingface/setfit/blob/6ee9b9d9efbffb2bcc364ba521007816ee89b6cf/src/setfit/trainer.py#L419

I am happy to make a PR for this, but the only way I can see to do it at the moment is to add at least one other argument to `SetFitTrainer`. It might also be useful to allow for the setting of arguments in evaluation functions, for example, `average` when calculating `roc_auc`. Given that, and if no one else can think of another way of doing it, this change should maybe wait for the outcome of https://github.com/huggingface/setfit/issues/179 and https://github.com/huggingface/setfit/issues/238.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.