huggingface / huggingface/setfit
Identity pairs are included in the training
- Dominant language
- Jupyter Notebook
- Stars
- 2.8k
- Forks
- 267
- Avg merge
- 36m
- Merged PRs (30d)
- 5
Description
Currently the training always includes identity pairs as `shuffle_combinations(iterable: Iterable, replacement: bool = True)` has the `replacement` parameter set by default to true. This is inconsistent with the setfit paper:
> Assuming that a small number (K) of labeled examples are given for a binary classification task, the
potential size of the ST fine-tuning set T is derived
from the number of unique sentence pairs that can
be generated, namely K(K − 1)/2, which is significantly larger than just K.
Putting aside the alignment with the paper, I think it does not make sense to include these pairs in the dataset as they are already "perfectly fitted".
Is this behaviour intentional? If yes, are there any plans for a possibility for switching the identity pairs inclusion off?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.