❓ [Question] torch_tensorrt.dynamo.compile hangs indefinitely mid compilation?

❓ Question

torch_tensorrt.dynamo.compile hangs indefinitely mid compilation cpu usage is through the roof and having debug = True shows that there's a step where it fails

What you have already tried

I tried compiling with torchscript and it works well enough but i wanted to test the dynamo backend

Environment

Python 3.9.2
torch 2.2+cu118
torch_tensorrt 2.2+cu118
tensorrt 8.6

Build information about Torch-TensorRT can be found by turning on debug messages

• PyTorch Version (e.g., 1.0): 2.2
• CPU Architecture: x86_64
• OS (e.g., Linux): debian 11
• How you installed PyTorch (conda, pip, libtorch, source): pip install torch torchvision torch_tensorrt --index-url https://download.pytorch.org/whl/cu118
• Build command you used (if compiling from source):

import torch
import torch_tensorrt
from gfpgan.archs.gfpganv1_clean_arch import GFPGANv1Clean

gfpgan = GFPGANv1Clean(
                out_size=512,
                num_style_feat=512,
                channel_multiplier=2,
                decoder_load_path=None,
                fix_decoder=False,
                num_mlp=8,
                input_is_latent=True,
                different_w=True,
                narrow=1,
                sft_half=True)

model_path="./experiments/pretrained_models/GFPGANv1.3.pth"
loadnet = torch.load(model_path)
if 'params_ema' in loadnet:
    keyname = 'params_ema'
else:
    keyname = 'params'
gfpgan.load_state_dict(loadnet[keyname], strict=True)
gfpgan = gfpgan.eval()
inputs=[torch.randn([8, 3, 512, 512],dtype=torch.float32).cuda()]

if torch.cuda.is_available():
    gfpgan = gfpgan.cuda().eval()
    torch.set_float32_matmul_precision('high')
    compiled = torch.compile(gfpgan,
                            backend="aot_torch_tensorrt_aten",
                            options={
                                "truncate_long_and_double":True,
                                "debug":True
                            })
    print("EXPORTING")
    import time
    start= time.time()
    print(compiled(*inputs))
    print(time.time()-start)
    torch.save(compiled, "compiled.ts")

• Are you using local sources or building from archives:
• Python version: 3.9.2
• CUDA version: 118 (12.3 installed on OS)
• GPU models and configuration: nvidia A100 80gb and nvidia L4 both have the same behavior
• Any other relevant information:
  private fork based on https://github.com/TencentARC/GFPGAN

Additional context