huggingface / huggingface/setfit

Hyperparameter tuning for AbsaModel

Open
#494 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
2.8k
Forks
267
Avg merge
36m
Merged PRs (30d)
5

Description

Hi! I would like to do hyperparameter search for an ABSA model (polarity only to be precise). Since there is no `hyperparameter_search` method for `AbsaTrainer`, I do this like in the code below, however, I am not certain if I do this right/optimally.

```
def objective(trial):
params = {
'batch_size': trial.suggest_categorical('batch_size', [8, 16, 32]),
'body_learning_rate': trial.suggest_float('learning_rate', 1e-5, 1e-3),
'span_contexts': trial.suggest_categorical('span_contexts', [(0, 3), (0, 4), (0, 5)])
}

model = AbsaModel.from_pretrained(
model_id='all-mpnet-base-v2',
spacy_model='en_core_web_sm',
span_contexts=params.pop('span_contexts')
)

trainer = AbsaTrainer(
model,
train_dataset=train,
metric=compute_metrics
)

trainer.train_polarity(args=TrainingArguments(**params))

res = model.predict(eval)

f1 = f1_score(res['label'].tolist(), res['pred_polarity'].tolist(), average='macro')

return f1

study = optuna.create_study(direction='maximize', sampler=optuna.samplers.TPESampler(seed=99))

study.optimize(objective, n_trials=100, gc_after_trial=True)
```

Some questions:
- Are sentence transformer weights re-initialized from scratch at the start of each trial? Should I set `force_download=True` in AbsaModel?
- When I only train polarity and try evaluating results with `trainer.evaluate(eval)`, I get the following error: ```NotFittedError: This LogisticRegression instance is not fitted yet. Call 'fit' with appropriate arguments before using this estimator.``` However, the model clearly has a trained LR head and model.predict(eval) works just fine.
- Less of a technical question, but would it make sense to try different multi-target strategies for a 3-way classification (positive, neutral, negative)? If so, would I be correct to instantiate classes inside `objective` like this:
```
lr_ = LinearRegression()
st_model_ = SentenceTransformer('all-mpnet-base-v2')
aspect_model = AspectModel(st_model_, lr_, spacy_model='en_core_web_sm')

aspect_extractor = AspectExtractor(spacy_model='en_core_web_sm')

def objective(trial):
params = {
'batch_size': trial.suggest_categorical('batch_size', [8, 16, 32]),
'body_learning_rate': trial.suggest_float('learning_rate', 1e-5, 1e-3),
'multi_target_strategy': trial.suggest_categorical('multi_target_strategy', ['one-vs-rest', 'multi-output', 'classifier-chain'])
}

st_model = SentenceTransformer('all-mpnet-base-v2')
lr = LinearRegression()
pol_model = PolarityModel(
st_model,
lr,
multi_target_strategy=params.pop('multi_target_strategy'),
spacy_model='en_core_web_sm')

model = AbsaModel(
aspect_extractor=aspect_extractor,
aspect_model=aspect_model,
polarity_model=pol_model
)

trainer = AbsaTrainer(
model,
train_dataset=train,
metric=compute_metrics
)

(...)

```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.