graykode / graykode/nlp-tutorial

The Adam in 5-1.Transformer should be replaced by SGD

未关闭
#76 0 条评论 4 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Jupyter Notebook
星标
14.9k
派生
3.9k
PR 合并指标
30 天内没有已合并 PR

描述

Line 202 :
`optimizer = optim.Adam(model.parameters(), lr=0.001)`

In practice, I think the effect of Adam is quite bad. When epoch = 10, cost is 1.6; when epoch = 100 or 1000, cost is still equal to 1.6.
So I think we can change Adam to SGD, that is, `optimizer = optim.SGD(model.parameters(), lr=0.001)`

Here are the effects of using SGD:
```
Epoch: 0100 cost = 0.047965
Epoch: 0200 cost = 0.020129
Epoch: 0300 cost = 0.012563
Epoch: 0400 cost = 0.009101
Epoch: 0500 cost = 0.007131
Epoch: 0600 cost = 0.005862
Epoch: 0700 cost = 0.004978
Epoch: 0800 cost = 0.004325
Epoch: 0900 cost = 0.003823
Epoch: 1000 cost = 0.003426
```

贡献指南

打开贡献指南

调研方向

Open the 5-1.Transformer notebook and inspect line 202, where the optimizer is configured. Run the training example before and after the requested optimizer change, and confirm that the cost decreases over later epochs rather than remaining at 1.6.

由索引模型根据 Issue 内容生成。

评估

技术栈
python, pytorch
领域
machine-learning
Issue 类型
缺陷
难度
1/5
预计耗时
1 小时以内
活跃度
停滞
描述清晰度
描述清楚
新手友好度
52/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。