ageron / ageron/handson-mlp

[bug] nan validation loss when finetuning BERT sentiment analysis classifier via Trainer API

未關閉
#36 0 則留言 0 個 reaction 已指派 1 人 已被 @ageron 認領 在 GitHub 檢視
bug help wanted
主要語言
Jupyter Notebook
星號
2k
分支
625
平均合併
7 小時 55 分鐘
30 天內合併 PR
1

描述

### Enter the chapter number

Chapter 14

### Enter the page number

_No response_

### What is the cell's number in the notebook

Cell 120

### Enter the environment you are using to run the notebook

Colab

### Describe your issue

I'm running chapter 14 in colab, specifically the sections "Task-specific classes" and "The Trainer API" (cells 112-120), and the trainer is reporting a `nan` validation loss with accuracy between 40-50%, i.e., something completely breaks.

It seemed very odd, since validation metric looks indeed correct and the loss is backed into the Trainer object itself.

### Enter what you expected to happen

_No response_

### If you found a workaround, describe it here

Since I was skeptical the issue was with the metric/loss, I checked the float16.

First I tried to remove the conversion, but the T4 of colab was forcing to extremely reduce the batch size making 1 epoch incredibly long. Then I realised that Trainer class offers flags to specify the model weights dtype.

Overall, I resolved the problem by commenting out the `dtype=torch.float16` in the instantiation of the pretrained model in cell 112

```
bert_for_binary_clf = BertForSequenceClassification.from_pretrained(
"bert-base-uncased",
num_labels=2,
#dtype=torch.float16 #<---- remove this
).to(device)
```
and adding type casting using the TrainerArguments options, based on the [official documentation](https://huggingface.co/docs/transformers/v5.4.0/en/main_classes/trainer#transformers.TrainingArguments.fp16) in cell 119

```
train_args = TrainingArguments(
output_dir="my_imdb_model", num_train_epochs=2,
per_device_train_batch_size=128,
per_device_eval_batch_size=128,
eval_strategy="epoch",
logging_strategy="epoch",
save_strategy="epoch",
load_best_model_at_end=True,
metric_for_best_model="accuracy",
report_to="none",
fp16=True. # <--- add this
)
```

With these two changes I get a validation loss of around 0.3 and (more important) the validation accuracy is around 0.9 (as originally in the notebook).

貢獻指南

這個儲存庫沒有索引到貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。