RuntimeError: expected scalar type Float but found Half

Open
#849 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
28/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
python, pytorch
Domain
machine-learning

Research direction

Start with pcdet/ops/iou3d_nms/iou3d_nms_utils.py, especially boxes_iou3d_gpu and its call to iou3d_nms_cuda.boxes_overlap_bev_gpu. Read the Apex AMP float_function documentation and inspect the custom operator boundary. Done means reproducing the reported mismatch and establishing the correct annotation or casting behavior for the operator.

Written by the indexing model from the issue text.

Description

I'm trying to use APEX AMP to speed up training time on 3D object detection model. But I get float type mismatch error on https://github.com/sshaoshuai/PCDet/blob/master/pcdet/ops/iou3d_nms/iou3d_nms_utils.py

  File "/usr/local/lib/python3.6/dist-packages/apex/amp/amp.py", line 22, in wrapper
    return orig_fn(*args, **kwargs)
  File "/opt/robotorch/project/part_a2/pcdet/ops/iou3d_nms/iou3d_nms_utils.py", line 48, in boxes_iou3d_gpu
    boxes_a_bev.contiguous(), boxes_b_bev.contiguous(), overlaps_bev
RuntimeError: expected scalar type Float but found Half (data_ptr<float> at /home/cruise/.local/lib/python3.6/site-packages/torch/include/ATen/core/TensorMethods.h:5747)

Within the function "boxes_iou3d_gpu", we have custom operator implementation on "boxes_overlap_bev_gpu".

iou3d_nms_cuda.boxes_overlap_bev_gpu(boxes_a_bev.contiguous(), boxes_b_bev.contiguous(), overlaps_bev)

I follow the instruction in https://github.com/NVIDIA/apex/tree/master/apex/amp to annotate user functions to cast all arguments to fp32, but the error stays the same:

@amp.float_function
def boxes_iou3d_gpu(boxes_a, boxes_b):

What is the correct way to use "float_function" annotation to cast all arguments to fp32?

Dominant language
Python
Stars
9k
Forks
1.5k
Avg merge
2d 4h
Merged PRs (30d)
3

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from NVIDIA/apex

All issues in NVIDIA/apex

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.