Segfault after tracing quantized mobilenet v3 using nightly version

🐛 Describe the bug

Sample code to reproduce:

import torch
import torchvision
from torchvision import transforms
from PIL import Image

def download_file(url, filename):
    import urllib
    try:
        urllib.URLopener().retrieve(url, filename)
    except:
        urllib.request.urlretrieve(url, filename)


def download_data():
    download_file("https://github.com/pytorch/hub/raw/master/images/dog.jpg", "dog.jpg")

def trace_mobilenet():
    model = torchvision.models.quantization.mobilenet_v3_large(pretrained=True, progress=True, quantize=True)
    model.eval()

    # validate that model runs
    input_image = Image.open("dog.jpg")
    preprocess = transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
    ])
    input_tensor = preprocess(input_image)
    input_batch = input_tensor.unsqueeze(0)  # create a mini-batch as expected by the model

    print('start tracing...')
    export_model = torch.jit.trace(model, input_batch)
    print('end tracing...')
    print(model)
    print(export_model)

download_data()
trace_mobilenet()

Output:

/anaconda3/envs/torch37/lib/python3.7/site-packages/torch/ao/quantization/utils.py:175: UserWarning: must run observer before calling calculate_qparams. Returning default values.
  "must run observer before calling calculate_qparams. " +
start tracing...
end tracing...
Segmentation fault (core dumped)

The segfault happens randomly. When I tried to debug using pdb, the segfault happens at different lines each time (original code tried to do other stuff after tracing, the above print code seems to be the simplest repro I could create), and they all happen after torch.jit.trace is called.

Versions

Collecting environment information...
PyTorch version: 1.11.0.dev20220127+cpu
Is debug build: False
CUDA used to build PyTorch: Could not collect
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.3 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: version 3.22.2
Libc version: glibc-2.9

Python version: 3.7.0 (default, Oct 9 2018, 10:31:47) [GCC 7.3.0] (64-bit runtime)
Python platform: Linux-5.13.0-27-generic-x86_64-with-debian-bullseye-sid
Is CUDA available: False
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: NVIDIA GeForce GTX 1080
Nvidia driver version: 510.39.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.3.2
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.3.2
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.3.2
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.3.2
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.3.2
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.3.2
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.3.2
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.21.2
[pip3] torch==1.11.0.dev20220127+cpu
[pip3] torchvision==0.12.0.dev20220127+cpu
[conda] mkl 2022.0.1 h06a4308_117
[conda] mkl-include 2022.0.1 h06a4308_117
[conda] numpy 1.21.2 py37hd8d4704_0
[conda] numpy-base 1.21.2 py37h2b8c604_0
[conda] torch 1.11.0.dev20220127+cpu pypi_0 pypi
[conda] torchvision 0.12.0.dev20220127+cpu pypi_0 pypi

Start with the supplied Python reproducer, focusing on torchvision.models.quantization.mobilenet_v3_large and the torch.jit.trace call; the payload names no repository file or test. Confirm the intermittent post-trace segmentation fault and identify a reliable failure boundary before determining what test or fix is needed.