AlibabaResearch / AlibabaResearch/AdvancedLiterateMachinery

采用推荐脚本训练,在3万步左右会出现Loss is nan, stopping training,又遇到的嘛?

Open
#161 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
1.8k
Forks
195
PR merge metrics
No merged PRs in 30d

Description

Loss is nan, stopping training
{'pt_loss': tensor(nan, device='cuda:0', grad_fn=), 'poly_loss': tensor(nan, device='cuda:0', grad_fn=), 'rec_loss': tensor(3.2648, device='cuda:0', grad_fn=)}

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.