fp16 report nan on mask rcnn
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 9k
- Forks
- 1.5k
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 3
Description
I use apex in the mask_rcnn_r50_fpn_1x.py mmdetection, and use pytorch roi layer.
- when I choose opt_level:
O1, report error:
from torchvision.ops import roi_align as tv_roi_align
File "/opt/anaconda3/lib/python3.7/site-packages/torchvision/__init__.py", line 1, in <module>
from torchvision import models
File "/opt/anaconda3/lib/python3.7/site-packages/torchvision/models/__init__.py", line 12, in <module>
from . import detection
File "/opt/anaconda3/lib/python3.7/site-packages/torchvision/models/detection/__init__.py", line 1, in <module>
from .faster_rcnn import *
File "/opt/anaconda3/lib/python3.7/site-packages/torchvision/models/detection/faster_rcnn.py", line 13, in <module>
from .rpn import AnchorGenerator, RPNHead, RegionProposalNetwork
File "/opt/anaconda3/lib/python3.7/site-packages/torchvision/models/detection/rpn.py", line 8, in <module>
from . import _utils as det_utils
File "/opt/anaconda3/lib/python3.7/site-packages/torchvision/models/detection/_utils.py", line 74, in <module>
@torch.jit.script
File "/opt/anaconda3/lib/python3.7/site-packages/torch/jit/__init__.py", line 1181, in script
return _compile_function(fn=obj, qualified_name=qualified_name, _frames_up=_frames_up + 1, _rcb=_rcb)
File "/opt/anaconda3/lib/python3.7/site-packages/torch/jit/__init__.py", line 1077, in _compile_function
script_fn = torch._C._jit_script_compile(qualified_name, ast, _rcb, get_default_args(fn))
File "/opt/anaconda3/lib/python3.7/site-packages/torch/jit/__init__.py", line 1009, in _try_compile_fn
qualified_name = _qualified_name(fn)
File "/opt/anaconda3/lib/python3.7/site-packages/torch/_jit_internal.py", line 404, in _qualified_name
"__module__ can't be None.".format(name))
RuntimeError: Could not get qualified name for class 'log': __module__ can't be None.
- when I choose opt_level:
O2, report error:
ValueError: nan or inf found in loss.
I set loss_scale to 1.0, the program also report this error.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the two configurations using mask_rcnn_r50_fpn_1x.py, Apex opt_level O1 and O2, and the torchvision.ops roi_align path mentioned in the traceback. Start by recording the PyTorch, torchvision, Apex, and mmdetection versions, then isolate whether the import failure and the NaN loss have separate causes. Done means identifying a reproducible cause and a verified fix or clear compatibility guidance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100