clab / clab/dynet

RuntimeError: Magnitude of gradient is bad: -nan

Open
#1,586 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
3.4k
Forks
701
PR merge metrics
No merged PRs in 30d

Description

I have a complex function with three parameters in total (of size 1 each).

Calculating the function using `.value()` gives me the desired output. At no point, I divide by 0, or multiply by infinity as far as I am aware.

However, in the first step of training, I always get:
> RuntimeError: Magnitude of gradient is bad: -nan

I narrowed it down to one of the parameters, which is in the denominator of a `dy.cdiv`, but it starts equaling `1`, so I don't understand how can that cause a `-nan` gradient.

For debugging, I am using:
```python
dy.renew_cg(immediate_compute=True, check_validity=True)
```

But that doesn't catch this before I run `trainer.update()`.

Is there a way to narrow down and figure out at what point in the network the `-nan` gradient happens exactly? Maybe if I know exactly where, I can understand what configuration is causing this.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the training step using dy.renew_cg(immediate_compute=True, check_validity=True), then inspect the computation involving dy.cdiv and the call to trainer.update(). Trace where the gradient becomes -nan and document the triggering configuration or operation, along with a focused reproduction if possible.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.