Torch-TensorRT Integration

🚀 The feature

pytorch-tensorrt hit release 1.0 (is actually 1.1 right now), but most of the models available are not out of the box convertable to it.

Motivation, pitch

Read feature description

Alternatives

Alternatives would be to convert to onnx, and then convert to tensorrt, which is exactly what torch-trt tries to avoid; This would also require more work, because somebody would have to make sure that models are onnx compatible and tensort-trt compatible, with the latter being a torchscript-supporting runtime.

Additional context

AFAIK, from some quick tests like:

    module = fcos_resnet50_fpn()
    generalized_rcnn_transform_max_size = module.transform.max_size
    inputs = torch_trt.Input(
        min_shape=[1, 3, 224, 224],
        opt_shape=[1, 3, 1080, 1920],
        max_shape=[
            1,
            3,
            generalized_rcnn_transform_max_size,
            generalized_rcnn_transform_max_size,
        ],
    )

    precisions = {torch.half} # doesn't really matter

    trt_module = torch_trt.compile(
        module=module, inputs=[inputs], enabled_precisions=precisions
    )

And looking into some blocking issue from tensorrt:

• [Feature] Adding support for collections as part of many inputs/outputs in TRTorch programs

... the first thing to do would be to remove self/state (self.a = b) mutations, on the forward method, and... almost everything will work out of the box?

For reference, from a quick attempt to port some fcos model, I found 2 places where this happens. I am not sure if this is required.

• While printing the warning statement
• And on anchor utils here

Start by reproducing the Torch-TensorRT compile example for fcos_resnet50_fpn(), then inspect the referenced mutation in torchvision/models/detection/fcos.py and torchvision/models/detection/anchor_utils.py. Review the linked TensorRT collections issue before deciding scope; done would mean the relevant torchvision models convert to Torch-TensorRT without the reported blocking behavior.