huggingface / huggingface/course
LLM Course chapter 3 mistakes in content
- Dominant language
- MDX
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 13m
- Merged PRs (30d)
- 1
Description
I found some content issues in chapter 3. Fine-Tuning a Pretrained Model.
1. Like issue #926 mentions, this "--upgrade" line needs to be added to all of the notebooks: %pip install --upgrade datasets
2. Three or four subsections have AdamW incorrectly imported from transformers in their notebooks, which doesn't work:
from transformers import AdamW, AutoModelForSequenceClassification, get_scheduler
AdamW should be imported from torch.optim like this:
from torch.optim import AdamW
from transformers import AutoModelForSequenceClassification, get_scheduler
3. In subsection 5, Understanding Learning Curves, the graph under "Accuracy Curves" is incorrect. It's another graph of a loss curve.
4. In subsection 5, Understanding Learning Curves, under "Solutions for erratic curves:" after "In the sample below, we lower the learning rate and increase the batch size." it has:
-learning_rate=1e-5,
+learning_rate=1e-4,
That is incorrectly showing an increase in the learning rate--the minus sign indicating replacing 0.00001 with plus sign 0.0001. 1e-4 is higher (greater) than 1e-5 so it is not "lowering" the learning rate. For example, increasing the learning rate from 1e-5 to 1e-4 means the model will take larger steps during training.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.