alibaba / alibaba/FederatedScope

WARNING: Skip the batch due to the loss is NaN, it may be caused by exceeding the precision or invalid labels

Open
#777 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.5k
Forks
261
PR merge metrics
No merged PRs in 30d

Description

When I run
"python federatedscope/main.py --cfg federatedscope/llm/baseline/exp_yaml/dolly_lda/dolly_federate.yaml"

why there usually exists
"WARNING: Skip the batch due to the loss is NaN, it may be caused by exceeding the precision or invalid labels"?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by running `python federatedscope/main.py --cfg federatedscope/llm/baseline/exp_yaml/dolly_lda/dolly_federate.yaml` and capture the batches that produce the warning. Trace where the warning is emitted, then inspect the configured labels and precision involved. Done means identifying the reproducible cause and documenting or correcting the conditions that trigger the NaN loss.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.