huggingface / huggingface/setfit
[Bug] `apply_hyperparameters` gives TypeError when `model_init` is not given
- Dominant language
- Jupyter Notebook
- Stars
- 2.8k
- Forks
- 267
- Avg merge
- 36m
- Merged PRs (30d)
- 5
Description
Hello!
## Bug overview
* `apply_hyperparameters` carelessly assumes that `model_init` was passed to the `Trainer` instance.
## How to Reproduce
Copy-pasteable reproducing script (Script 1)
```python
from datasets import load_dataset
from sentence_transformers.losses import CosineSimilarityLoss
from setfit import SetFitModel, SetFitTrainer, sample_dataset
# Load a dataset from the Hugging Face Hub
dataset = load_dataset("sst2")
# Simulate the few-shot regime by sampling 8 examples per class
train_dataset = sample_dataset(dataset["train"], label_column="label", num_samples=8)
eval_dataset = dataset["validation"]
# Load a SetFit model from Hub
model: SetFitModel = SetFitModel.from_pretrained(
"sentence-transformers/paraphrase-mpnet-base-v2",
)
# Create trainer
trainer = SetFitTrainer(
model=model,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
loss_class=CosineSimilarityLoss,
metric="accuracy",
batch_size=16,
num_iterations=20, # The number of text pairs to generate for contrastive learning
num_epochs=1, # The number of epochs to use for constrastive learning
column_mapping={"sentence": "text", "label": "label"} # Map dataset columns to text/label expected by trainer
)
trainer.apply_hyperparameters({"lr": 1e-5})
```
Running this script gives the following error:
```python
Traceback (most recent call last):
File "[sic]\setfit\demo_apply_hyperparams.py", line 31, in
trainer.apply_hyperparameters({"lr": 1e-5})
File "[sic]\setfit\src\setfit\trainer.py", line 184, in apply_hyperparameters
elif number_of_arguments(self.model_init) == 0: # we do not warn if model_init could be using it
File "[sic]\envs\setfit\lib\site-packages\transformers\trainer_utils.py", line 610, in number_of_arguments
return len(inspect.signature(func).parameters)
File "[sic]\envs\setfit\lib\inspect.py", line 3113, in signature
return Signature.from_callable(obj, follow_wrapped=follow_wrapped)
File "[sic]\envs\setfit\lib\inspect.py", line 2862, in from_callable
return _signature_from_callable(obj, sigcls=cls,
File "[sic]\envs\setfit\lib\inspect.py", line 2261, in _signature_from_callable
raise TypeError('{!r} is not a callable object'.format(obj))
TypeError: None is not a callable object
```
This is due to line 184 in the following snippet. If instead I passed an empty dictionary as params, then it would have crashed on line 190 as well.
https://github.com/huggingface/setfit/blob/4a613b08267690dad0840444fc3e2caf60f29a44/src/setfit/trainer.py#L170-L192
## Expected behaviour
I would expect either of the two scenarios:
* If we want to support `apply_hyperparameters` when a `model` is given to the Trainer, then I would expect it to actually set the model's hyperparameters.
* A warning that `apply_hyperparameters` cannot be used unless `model_init` is set.
## What now?
We should discuss which of the two expected behaviours, or another behaviour, we would like to see in practice.
- Tom Aarsen
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.