huggingface / huggingface/setfit
Hyperparameter tuning for AbsaModel
- Dominant language
- Jupyter Notebook
- Stars
- 2.8k
- Forks
- 267
- Avg merge
- 36m
- Merged PRs (30d)
- 5
Description
Hi! I would like to do hyperparameter search for an ABSA model (polarity only to be precise). Since there is no `hyperparameter_search` method for `AbsaTrainer`, I do this like in the code below, however, I am not certain if I do this right/optimally.
```
def objective(trial):
params = {
'batch_size': trial.suggest_categorical('batch_size', [8, 16, 32]),
'body_learning_rate': trial.suggest_float('learning_rate', 1e-5, 1e-3),
'span_contexts': trial.suggest_categorical('span_contexts', [(0, 3), (0, 4), (0, 5)])
}
model = AbsaModel.from_pretrained(
model_id='all-mpnet-base-v2',
spacy_model='en_core_web_sm',
span_contexts=params.pop('span_contexts')
)
trainer = AbsaTrainer(
model,
train_dataset=train,
metric=compute_metrics
)
trainer.train_polarity(args=TrainingArguments(**params))
res = model.predict(eval)
f1 = f1_score(res['label'].tolist(), res['pred_polarity'].tolist(), average='macro')
return f1
study = optuna.create_study(direction='maximize', sampler=optuna.samplers.TPESampler(seed=99))
study.optimize(objective, n_trials=100, gc_after_trial=True)
```
Some questions:
- Are sentence transformer weights re-initialized from scratch at the start of each trial? Should I set `force_download=True` in AbsaModel?
- When I only train polarity and try evaluating results with `trainer.evaluate(eval)`, I get the following error: ```NotFittedError: This LogisticRegression instance is not fitted yet. Call 'fit' with appropriate arguments before using this estimator.``` However, the model clearly has a trained LR head and model.predict(eval) works just fine.
- Less of a technical question, but would it make sense to try different multi-target strategies for a 3-way classification (positive, neutral, negative)? If so, would I be correct to instantiate classes inside `objective` like this:
```
lr_ = LinearRegression()
st_model_ = SentenceTransformer('all-mpnet-base-v2')
aspect_model = AspectModel(st_model_, lr_, spacy_model='en_core_web_sm')
aspect_extractor = AspectExtractor(spacy_model='en_core_web_sm')
def objective(trial):
params = {
'batch_size': trial.suggest_categorical('batch_size', [8, 16, 32]),
'body_learning_rate': trial.suggest_float('learning_rate', 1e-5, 1e-3),
'multi_target_strategy': trial.suggest_categorical('multi_target_strategy', ['one-vs-rest', 'multi-output', 'classifier-chain'])
}
st_model = SentenceTransformer('all-mpnet-base-v2')
lr = LinearRegression()
pol_model = PolarityModel(
st_model,
lr,
multi_target_strategy=params.pop('multi_target_strategy'),
spacy_model='en_core_web_sm')
model = AbsaModel(
aspect_extractor=aspect_extractor,
aspect_model=aspect_model,
polarity_model=pol_model
)
trainer = AbsaTrainer(
model,
train_dataset=train,
metric=compute_metrics
)
(...)
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.