huggingface / huggingface/setfit

Why are the models fine-tuned with CosineSimilarity between 0 and 1?

Open
#254 10 comments 4 reactions 0 assignees View on GitHub
question
Dominant language
Jupyter Notebook
Stars
2.8k
Forks
267
Avg merge
36m
Merged PRs (30d)
5

Description

Hi everyone,

This is a small question related to how models are fine-tuned during the first step of training. I see that the default loss function is `losses.CosineSimilarityLoss`. But when generating sentence pairs [here](https://github.com/huggingface/setfit/blob/35c0511fa9917e653df50cb95a22105b397e14c0/src/setfit/modeling.py#L546), negative ones are assigned a 0 label.
I understand that having scores between 0 and 1 is ideal, because they can be interpreted as probabilities. But cosine similarity ranges from -1 to 1, so shouldn't we expect the full range to be used? The model head can then make predictions on a more isotropic embedding space.
Is this related to how Sentence Transformers are pre-trained?

Thanks for your clarifications!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.