NVIDIA / NVIDIA/apex

fp16 report nan on mask rcnn

Open
#685 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
9k
Forks
1.5k
Avg merge
2d 4h
Merged PRs (30d)
3

Description

I use apex in the mask_rcnn_r50_fpn_1x.py mmdetection, and use pytorch roi layer.

  1. when I choose opt_level: O1, report error:
  from torchvision.ops import roi_align as tv_roi_align
  File "/opt/anaconda3/lib/python3.7/site-packages/torchvision/__init__.py", line 1, in <module>
    from torchvision import models
  File "/opt/anaconda3/lib/python3.7/site-packages/torchvision/models/__init__.py", line 12, in <module>
    from . import detection
  File "/opt/anaconda3/lib/python3.7/site-packages/torchvision/models/detection/__init__.py", line 1, in <module>
    from .faster_rcnn import *
  File "/opt/anaconda3/lib/python3.7/site-packages/torchvision/models/detection/faster_rcnn.py", line 13, in <module>
    from .rpn import AnchorGenerator, RPNHead, RegionProposalNetwork
  File "/opt/anaconda3/lib/python3.7/site-packages/torchvision/models/detection/rpn.py", line 8, in <module>
    from . import _utils as det_utils
  File "/opt/anaconda3/lib/python3.7/site-packages/torchvision/models/detection/_utils.py", line 74, in <module>
    @torch.jit.script
  File "/opt/anaconda3/lib/python3.7/site-packages/torch/jit/__init__.py", line 1181, in script
    return _compile_function(fn=obj, qualified_name=qualified_name, _frames_up=_frames_up + 1, _rcb=_rcb)
  File "/opt/anaconda3/lib/python3.7/site-packages/torch/jit/__init__.py", line 1077, in _compile_function
    script_fn = torch._C._jit_script_compile(qualified_name, ast, _rcb, get_default_args(fn))
  File "/opt/anaconda3/lib/python3.7/site-packages/torch/jit/__init__.py", line 1009, in _try_compile_fn
    qualified_name = _qualified_name(fn)
  File "/opt/anaconda3/lib/python3.7/site-packages/torch/_jit_internal.py", line 404, in _qualified_name
    "__module__ can't be None.".format(name))
RuntimeError: Could not get qualified name for class 'log': __module__ can't be None.
  1. when I choose opt_level: O2, report error:
ValueError: nan or inf found in loss.

I set loss_scale to 1.0, the program also report this error.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the two configurations using mask_rcnn_r50_fpn_1x.py, Apex opt_level O1 and O2, and the torchvision.ops roi_align path mentioned in the traceback. Start by recording the PyTorch, torchvision, Apex, and mmdetection versions, then isolate whether the import failure and the NaN loss have separate causes. Done means identifying a reproducible cause and a verified fix or clear compatibility guidance.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.