huggingface / huggingface/setfit
Support metrics using prediction_scores in `SetFitTrainer.evaluate`
- Dominant language
- Jupyter Notebook
- Stars
- 2.8k
- Forks
- 267
- Avg merge
- 36m
- Merged PRs (30d)
- 5
Description
Can I make a request for `SetFitTrainer.evaluate` to allow the calculation of metrics that take prediction scores as input, i.e., the output of `self.model.predict_proba`? The relevant line is this one: https://github.com/huggingface/setfit/blob/6ee9b9d9efbffb2bcc364ba521007816ee89b6cf/src/setfit/trainer.py#L419
I am happy to make a PR for this, but the only way I can see to do it at the moment is to add at least one other argument to `SetFitTrainer`. It might also be useful to allow for the setting of arguments in evaluation functions, for example, `average` when calculating `roc_auc`. Given that, and if no one else can think of another way of doing it, this change should maybe wait for the outcome of https://github.com/huggingface/setfit/issues/179 and https://github.com/huggingface/setfit/issues/238.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.