RuntimeError: Magnitude of gradient is bad: -nan
- Dominant language
- C++
- Stars
- 3.4k
- Forks
- 701
- PR merge metrics
- No merged PRs in 30d
Description
I have a complex function with three parameters in total (of size 1 each).
Calculating the function using `.value()` gives me the desired output. At no point, I divide by 0, or multiply by infinity as far as I am aware.
However, in the first step of training, I always get:
> RuntimeError: Magnitude of gradient is bad: -nan
I narrowed it down to one of the parameters, which is in the denominator of a `dy.cdiv`, but it starts equaling `1`, so I don't understand how can that cause a `-nan` gradient.
For debugging, I am using:
```python
dy.renew_cg(immediate_compute=True, check_validity=True)
```
But that doesn't catch this before I run `trainer.update()`.
Is there a way to narrow down and figure out at what point in the network the `-nan` gradient happens exactly? Maybe if I know exactly where, I can understand what configuration is causing this.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the training step using dy.renew_cg(immediate_compute=True, check_validity=True), then inspect the computation involving dy.cdiv and the call to trainer.update(). Trace where the gradient becomes -nan and document the triggering configuration or operation, along with a focused reproduction if possible.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100