Megvii-BaseDetection / Megvii-BaseDetection/YOLOX
训练自己的数据集要么爆显存要么梯度爆炸
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 10.6k
- Forks
- 2.5k
- PR merge metrics
- No merged PRs in 30d
Description
4张V100训练自己的数据集（11762张图，前后景二分类检测），一段时间后报错显存不够，或者混合精度训练的scale减小到接近于0，loss上升且map变为0。之前训练其他比较小的数据集能够正常收敛，在这个数据集上一直训练不成功，尝试了减小batch size和学习率，都没有解决。同样的数据集在mmdet2.0上用faster rcnn训练是正常的，虽然中间有些iter会变成nan，但是后续会正常收敛。请问是和混合精度训练有关吗？尝试关闭混合训练会在一个iter上面停留很久。
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source file, configuration, or failing test is identified. Start by reproducing the custom-dataset run with mixed precision enabled, then compare memory use, loss, scale, and mAP against the reported behavior; the issue is done when training avoids out-of-memory or exploding gradients and converges reliably.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- computer-vision, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100