Lightning-AI / Lightning-AI/lit-llama
[question] nan loss value and run time error
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 6.1k
- Forks
- 517
- PR merge metrics
- No merged PRs in 30d
Description
Dear all,
I finetuning mydata with open llama. While running finetune/lora.py, I don't see the loss number as follows:
...
iter 3198: loss nan, time: 134.94ms
and while validating, it ends up with an error:
...
File ".../lit-llama/generate.py", line 74, in generate
idx_next = torch.multinomial(probs, num_samples=1).to(dtype=dtype)
RuntimeError: probability tensor contains either `inf`, `nan` or element < 0
Could you tell me how I can resolve this?
Thanks in advance.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reported NaN loss while running finetune/lora.py, then inspect generate.py at line 74 where torch.multinomial fails. Determine the input conditions that produce invalid probabilities during validation; done means the cause is isolated and a verified resolution is documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100