huggingface / huggingface/setfit

Identity pairs are included in the training

Open
#620 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
2.8k
Forks
267
Avg merge
36m
Merged PRs (30d)
5

Description

Currently the training always includes identity pairs as `shuffle_combinations(iterable: Iterable, replacement: bool = True)` has the `replacement` parameter set by default to true. This is inconsistent with the setfit paper:

> Assuming that a small number (K) of labeled examples are given for a binary classification task, the
potential size of the ST fine-tuning set T is derived
from the number of unique sentence pairs that can
be generated, namely K(K − 1)/2, which is significantly larger than just K.

Putting aside the alignment with the paper, I think it does not make sense to include these pairs in the dataset as they are already "perfectly fitted".
Is this behaviour intentional? If yes, are there any plans for a possibility for switching the identity pairs inclusion off?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.