ageron / ageron/handson-mlp

[bug] OneCycleLR scheduler should be called after every batch

未關閉
#28 0 則留言 0 個 reaction 已指派 1 人 已被 @ageron 認領 在 GitHub 檢視
bug help wanted
主要語言
Jupyter Notebook
星號
2k
分支
625
平均合併
7 小時 55 分鐘
30 天內合併 PR
1

描述

### Enter the chapter number

Chapter 11. Training Deep Neural Networks

### Enter the page number

_No response_

### What is the cell's number in the notebook

Cells 105-106 in 11_training_deep_neural_networks.ipynb

### Enter the environment you are using to run the notebook

Jupyter on MacOS

### Describe your issue

The last step of Exercise 8 reads:

Step 7: _Retrain your model using 1cycle scheduling and see if it improves training speed and model accuracy._

Solution code in cells 105 and 106:
```python
n_epochs = 60
optimizer = torch.optim.NAdam(model.parameters(), lr=1e-3)
scheduler = torch.optim.lr_scheduler.OneCycleLR(
optimizer, epochs=n_epochs, steps_per_epoch=len(train_loader), max_lr=1e-2)
criterion = nn.CrossEntropyLoss()
accuracy = torchmetrics.Accuracy(task="multiclass", num_classes=10).to(device)
```
```python
history = train_with_early_stopping(model, optimizer, criterion, accuracy,
train_loader, valid_loader, n_epochs,
patience=20, scheduler=scheduler)
```

_train_with_early_stopping()_ function (defined earlier in the notebook) calls `scheduler.step()` at the end of every epoch. This seems to work, however, the [documentation of OneCycleLR scheduler ](https://docs.pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.OneCycleLR.html)mentions that it should be called at the end of every batch.

Also, using NAdam optimizer resulted in validation accuracy consistently dropping to ~0.10 after a few epochs and training diverging, although this might be an issue with my environment. I resolved this by replacing NAdam with SGD optimizer.

### Enter what you expected to happen

_No response_

### If you found a workaround, describe it here

_No response_

貢獻指南

這個儲存庫沒有索引到貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。