Megvii-BaseDetection / Megvii-BaseDetection/DynamicRouting
FloatingPointError: Loss became infinite or NaN at iteration=5593
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 380
- Forks
- 45
- PR merge metrics
- No merged PRs in 30d
Description
It seems that the loss is too large. I used only 1 gpu to train and set batchsize to 1, and I did not change any other configs. I have tried to set learning rate to 1e-3 and 1e-4, but this error still happen.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named. Start by reproducing training with one GPU, batch size 1, and the reported learning rates, then inspect the training and loss behavior near iteration 5593; done means the NaN or infinite loss has an identified cause and a verified resolution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100