Pytorch的SGD，Adam和RMSprop的分析和复现 | 小王同学

https://a171232886.github.io/2021/12/29/Pytorch%E7%9A%84SGD%EF%BC%8CAdam%E5%92%8CRMSprop%E7%9A%84%E5%88%86%E6%9E%90%E5%92%8C%E5%A4%8D%E7%8E%B0/

前言
我总觉得，书本上说的是一回事，实际是另一回事。应实际看看优化器到底怎么算的，用矩阵运算把结果复现出来。
SGD
官方文档
随机梯度下降(Stochastic Gradient Descent)，这要强调和GD的区别。书本上的GD是遍历完所有的样本之后进行一次梯度下降，SGD是在得到一个样本后进行一次梯度下降，mini-batch是进行一定数量的样本之后才进行一次梯度下降。而我们实际使

The issue links to a blog post analyzing and reimplementing SGD, Adam, and RMSprop optimizers in PyTorch. Start by reading the linked article to understand the mathematical derivations and implementation steps. Then, examine PyTorch's optimizer source code (likely in torch/optim/) to compare with the custom implementations. The goal is to verify the reimplementation matches PyTorch's behavior, possibly by writing small test scripts.

Scritto dal modello di indicizzazione a partire dal testo della issue.