Gradient clipping is not consistent with other frameworks
- Dominant language
- C++
- Stars
- 3.4k
- Forks
- 701
- PR merge metrics
- No merged PRs in 30d
Description
Right now dynet clips gradients separately for each parameters. This means that if you clip your gradients to 1.0 but have 16 `Parameter` objects, the total gradient will be clipped to a norm of 4.
The correct thing to do would be to clip the total norm, like pytorch does https://pytorch.org/docs/master/_modules/torch/nn/utils/clip_grad.html
I've found that it makes a difference (in LM experiments)
Contributor guide
No contributing guide indexed for this repository
Research direction
No file or test is named. Locate DyNet's gradient-clipping entry point, then compare its behavior with the linked PyTorch implementation. Reproduce the multi-Parameter case described in the issue and verify that clipping is based on the total gradient norm rather than each parameter separately.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100