NaN or Inf detected when computing log_softmax
- Dominant language
- C++
- Stars
- 3.4k
- Forks
- 701
- PR merge metrics
- No merged PRs in 30d
Description
Immediately when I start training a model I get "NaN of Inf detected" when this line happens:
`logloss = log_softmax(f_i, valid_frames)`
Note this is with immediate_compute and check_validity turned on. If they aren't, then the error seems to happen a little later in the process.
In the most recent run, the values being passed to log_softmax are:
f_i = expression 1630/2
valid_frames = [204, 28]
Can someone help me understand why this input is returning either inf or nan? I've looked through the issues and it seems to be something different each time.
Here is an example of what log_softmax returns (not the same run as above though):
logloss = expression 3429/2
Thanks!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the log_softmax entry point and reproduce the failure with immediate_compute and check_validity enabled, using the reported valid_frames value and a representative f_i expression. Trace where the NaN or Inf is first detected, compare behavior when those options are disabled, and document a reliable cause and reproduction case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100