huggingface / huggingface/course

LLM Course chapter 3 mistakes in content

Open
#980 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
MDX
Stars
4.2k
Forks
1.4k
Avg merge
13m
Merged PRs (30d)
1

Description

I found some content issues in chapter 3. Fine-Tuning a Pretrained Model.

1. Like issue #926 mentions, this "--upgrade" line needs to be added to all of the notebooks: %pip install --upgrade datasets

2. Three or four subsections have AdamW incorrectly imported from transformers in their notebooks, which doesn't work:

from transformers import AdamW, AutoModelForSequenceClassification, get_scheduler

AdamW should be imported from torch.optim like this:

from torch.optim import AdamW
from transformers import AutoModelForSequenceClassification, get_scheduler

3. In subsection 5, Understanding Learning Curves, the graph under "Accuracy Curves" is incorrect. It's another graph of a loss curve.

4. In subsection 5, Understanding Learning Curves, under "Solutions for erratic curves:" after "In the sample below, we lower the learning rate and increase the batch size." it has:

-learning_rate=1e-5,
+learning_rate=1e-4,

That is incorrectly showing an increase in the learning rate--the minus sign indicating replacing 0.00001 with plus sign 0.0001. 1e-4 is higher (greater) than 1e-5 so it is not "lowering" the learning rate. For example, increasing the learning rate from 1e-5 to 1e-4 means the model will take larger steps during training.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.