deepspeedai / deepspeedai/DeepSpeed
[QUESTION] Skipping examples with NaN loss
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
I'm trying to train a model on multiple GPUs with fp16 mixed precision and DeepSpeed + PyTorch Lightning. Certain pathological examples in the training set cause activations to overflow and ultimately the loss to become NaN. Is there any way to skip the backward pass for such examples? Alternatively, how is this problem usually addressed? My current hack is just to zero the loss in these cases, but that interferes with average loss computation and is generally inelegant.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files or tests are identified. Start by locating the DeepSpeed training path used with PyTorch Lightning, multi-GPU execution, and fp16 mixed precision, then reproduce the NaN-loss case described here. Done would require an agreed way to handle pathological examples without corrupting average-loss computation, with coverage for distributed training.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- machine-learning, python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100