ageron / ageron/handson-mlp

[bug] nan validation loss when finetuning BERT sentiment analysis classifier via Trainer API

Offen
#36 0 Kommentare 0 Reaktionen 1 zugewiesene Person Beansprucht von @ageron Auf GitHub ansehen
bug help wanted
Vorherrschende Sprache
Jupyter Notebook
Sterne
2k
Forks
624
Ø Merge
7 Std. 55 Min.
Gemergte PRs (30 T.)
1

Beschreibung

### Enter the chapter number

Chapter 14

### Enter the page number

_No response_

### What is the cell's number in the notebook

Cell 120

### Enter the environment you are using to run the notebook

Colab

### Describe your issue

I'm running chapter 14 in colab, specifically the sections "Task-specific classes" and "The Trainer API" (cells 112-120), and the trainer is reporting a `nan` validation loss with accuracy between 40-50%, i.e., something completely breaks.

It seemed very odd, since validation metric looks indeed correct and the loss is backed into the Trainer object itself.

### Enter what you expected to happen

_No response_

### If you found a workaround, describe it here

Since I was skeptical the issue was with the metric/loss, I checked the float16.

First I tried to remove the conversion, but the T4 of colab was forcing to extremely reduce the batch size making 1 epoch incredibly long. Then I realised that Trainer class offers flags to specify the model weights dtype.

Overall, I resolved the problem by commenting out the `dtype=torch.float16` in the instantiation of the pretrained model in cell 112

```
bert_for_binary_clf = BertForSequenceClassification.from_pretrained(
"bert-base-uncased",
num_labels=2,
#dtype=torch.float16 #<---- remove this
).to(device)
```
and adding type casting using the TrainerArguments options, based on the [official documentation](https://huggingface.co/docs/transformers/v5.4.0/en/main_classes/trainer#transformers.TrainingArguments.fp16) in cell 119

```
train_args = TrainingArguments(
output_dir="my_imdb_model", num_train_epochs=2,
per_device_train_batch_size=128,
per_device_eval_batch_size=128,
eval_strategy="epoch",
logging_strategy="epoch",
save_strategy="epoch",
load_best_model_at_end=True,
metric_for_best_model="accuracy",
report_to="none",
fp16=True. # <--- add this
)
```

With these two changes I get a validation loss of around 0.3 and (more important) the validation accuracy is around 0.9 (as originally in the notebook).

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.