clab / clab/dynet

NaN or Inf detected when computing log_softmax

Open
#1,634 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
3.4k
Forks
701
PR merge metrics
No merged PRs in 30d

Description

Immediately when I start training a model I get "NaN of Inf detected" when this line happens:

`logloss = log_softmax(f_i, valid_frames)`

Note this is with immediate_compute and check_validity turned on. If they aren't, then the error seems to happen a little later in the process.

In the most recent run, the values being passed to log_softmax are:

f_i = expression 1630/2
valid_frames = [204, 28]

Can someone help me understand why this input is returning either inf or nan? I've looked through the issues and it seems to be something different each time.

Here is an example of what log_softmax returns (not the same run as above though):

logloss = expression 3429/2

Thanks!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the log_softmax entry point and reproduce the failure with immediate_compute and check_validity enabled, using the reported valid_frames value and a representative f_i expression. Trace where the NaN or Inf is first detected, compare behavior when those options are disabled, and document a reliable cause and reproduction case.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.