ageron / ageron/handson-mlp

[bug] OneCycleLR scheduler should be called after every batch

Ouverte
#28 0 commentaires 0 réactions 1 personne assignée Réclamée par @ageron Voir sur GitHub
bug help wanted
Langage dominant
Jupyter Notebook
Étoiles
2k
Forks
624
Merge moyen
7 h 55 min
PR mergées (30 j)
1

Description

### Enter the chapter number

Chapter 11. Training Deep Neural Networks

### Enter the page number

_No response_

### What is the cell's number in the notebook

Cells 105-106 in 11_training_deep_neural_networks.ipynb

### Enter the environment you are using to run the notebook

Jupyter on MacOS

### Describe your issue

The last step of Exercise 8 reads:

Step 7: _Retrain your model using 1cycle scheduling and see if it improves training speed and model accuracy._

Solution code in cells 105 and 106:
```python
n_epochs = 60
optimizer = torch.optim.NAdam(model.parameters(), lr=1e-3)
scheduler = torch.optim.lr_scheduler.OneCycleLR(
optimizer, epochs=n_epochs, steps_per_epoch=len(train_loader), max_lr=1e-2)
criterion = nn.CrossEntropyLoss()
accuracy = torchmetrics.Accuracy(task="multiclass", num_classes=10).to(device)
```
```python
history = train_with_early_stopping(model, optimizer, criterion, accuracy,
train_loader, valid_loader, n_epochs,
patience=20, scheduler=scheduler)
```

_train_with_early_stopping()_ function (defined earlier in the notebook) calls `scheduler.step()` at the end of every epoch. This seems to work, however, the [documentation of OneCycleLR scheduler ](https://docs.pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.OneCycleLR.html)mentions that it should be called at the end of every batch.

Also, using NAdam optimizer resulted in validation accuracy consistently dropping to ~0.10 after a few epochs and training diverging, although this might be an issue with my environment. I resolved this by replacing NAdam with SGD optimizer.

### Enter what you expected to happen

_No response_

### If you found a workaround, describe it here

_No response_

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.