huggingface / huggingface/setfit

Clarification on end_to_end vs trainer.train_embeddings

Open
#529 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
2.8k
Forks
267
Avg merge
36m
Merged PRs (30d)
5

Description

I'm experimenting with a simple model right now and I'm confused about whether or not I should expect the sentence transformer model to change during the training process.

```python
# Define the model and training arguments
model = SetFitModel.from_pretrained(
"sentence-transformers/all-MiniLM-L6-v2",
multi_target_strategy="one-vs-rest",
use_differentiable_head=True,
head_params={"out_features": len(labels)},
labels=labels,
)

args = TrainingArguments(
batch_size=128,
# end_to_end=False,
# body_learning_rate=10.0,
num_epochs=4,
evaluation_strategy="no",
save_strategy="no",
load_best_model_at_end=True,
)

trainer = Trainer(
model=model,
args=args,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
metric="accuracy",
column_mapping={
"text": "text",
"label": "label",
}, # Map dataset columns to text/label expected by trainer
)
```

The documentation for `end_to_end` implies that the only time that the underlying model will change is when this argument is set, but experimentally that isn't true. The underlying sentence transformer (the "body" as I understand it) seems to always be trained in the `train()` logic, which is hard coded to always call `train_embeddings()`. I determined that the body changed by comparing the output scores of my model as well as the embeddings generated by the base sentence transformer model and the one that is set to my model body after training.

Did I misunderstand the docs? The only way I can get this to not happen is to comment out the `train_embeddings()` call in the setlib library's `train()` [here](https://github.com/huggingface/setfit/blob/327a3b64ca38dd688a7f92b67e7bf88f9db66aa9/src/setfit/trainer.py#L410)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.