google-research / google-research/sam
Implementation details of grad_norm computation
Open
- Dominant language
- Python
- Stars
- 646
- Forks
- 81
- PR merge metrics
- No merged PRs in 30d
Description
Hi guys, I am trying to implement SAM in MXNet and encounter two questions about grad_norm computation. When we compute SAM gradient e_w, should we calculate grad norm for each parameter, or just calculate the grad norm for all parameters at once? Hope for your advice.
Contributor guide
Assessment
This issue has not been assessed yet.