huggingface / huggingface/setfit

[Bug] `apply_hyperparameters` gives TypeError when `model_init` is not given

Open
#188 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
2.8k
Forks
267
Avg merge
36m
Merged PRs (30d)
5

Description

Hello!

## Bug overview
* `apply_hyperparameters` carelessly assumes that `model_init` was passed to the `Trainer` instance.

## How to Reproduce
Copy-pasteable reproducing script (Script 1)

```python
from datasets import load_dataset
from sentence_transformers.losses import CosineSimilarityLoss

from setfit import SetFitModel, SetFitTrainer, sample_dataset

# Load a dataset from the Hugging Face Hub
dataset = load_dataset("sst2")

# Simulate the few-shot regime by sampling 8 examples per class
train_dataset = sample_dataset(dataset["train"], label_column="label", num_samples=8)
eval_dataset = dataset["validation"]

# Load a SetFit model from Hub
model: SetFitModel = SetFitModel.from_pretrained(
"sentence-transformers/paraphrase-mpnet-base-v2",
)

# Create trainer
trainer = SetFitTrainer(
model=model,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
loss_class=CosineSimilarityLoss,
metric="accuracy",
batch_size=16,
num_iterations=20, # The number of text pairs to generate for contrastive learning
num_epochs=1, # The number of epochs to use for constrastive learning
column_mapping={"sentence": "text", "label": "label"} # Map dataset columns to text/label expected by trainer
)

trainer.apply_hyperparameters({"lr": 1e-5})
```

Running this script gives the following error:

```python
Traceback (most recent call last):
File "[sic]\setfit\demo_apply_hyperparams.py", line 31, in
trainer.apply_hyperparameters({"lr": 1e-5})
File "[sic]\setfit\src\setfit\trainer.py", line 184, in apply_hyperparameters
elif number_of_arguments(self.model_init) == 0: # we do not warn if model_init could be using it
File "[sic]\envs\setfit\lib\site-packages\transformers\trainer_utils.py", line 610, in number_of_arguments
return len(inspect.signature(func).parameters)
File "[sic]\envs\setfit\lib\inspect.py", line 3113, in signature
return Signature.from_callable(obj, follow_wrapped=follow_wrapped)
File "[sic]\envs\setfit\lib\inspect.py", line 2862, in from_callable
return _signature_from_callable(obj, sigcls=cls,
File "[sic]\envs\setfit\lib\inspect.py", line 2261, in _signature_from_callable
raise TypeError('{!r} is not a callable object'.format(obj))
TypeError: None is not a callable object
```

This is due to line 184 in the following snippet. If instead I passed an empty dictionary as params, then it would have crashed on line 190 as well.
https://github.com/huggingface/setfit/blob/4a613b08267690dad0840444fc3e2caf60f29a44/src/setfit/trainer.py#L170-L192

## Expected behaviour
I would expect either of the two scenarios:
* If we want to support `apply_hyperparameters` when a `model` is given to the Trainer, then I would expect it to actually set the model's hyperparameters.
* A warning that `apply_hyperparameters` cannot be used unless `model_init` is set.

## What now?
We should discuss which of the two expected behaviours, or another behaviour, we would like to see in practice.

- Tom Aarsen

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.