clab / clab/dynet

Gradient clipping is not consistent with other frameworks

Open
#1,374 2 comments 0 reactions 0 assignees View on GitHub
need more info
Dominant language
C++
Stars
3.4k
Forks
701
PR merge metrics
No merged PRs in 30d

Description

Right now dynet clips gradients separately for each parameters. This means that if you clip your gradients to 1.0 but have 16 `Parameter` objects, the total gradient will be clipped to a norm of 4.

The correct thing to do would be to clip the total norm, like pytorch does https://pytorch.org/docs/master/_modules/torch/nn/utils/clip_grad.html

I've found that it makes a difference (in LM experiments)

Contributor guide

No contributing guide indexed for this repository

Research direction

No file or test is named. Locate DyNet's gradient-clipping entry point, then compare its behavior with the linked PyTorch implementation. Reproduce the multi-Parameter case described in the issue and verify that clipping is based on the total gradient norm rather than each parameter separately.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.