Traced model output does not match the output of the original model

## Instructions To Reproduce the Issue:

Check https://stackoverflow.com/help/minimal-reproducible-example for how to ask good questions.
Simplify the steps to reproduce the issue using suggestions from the above link, and provide them below:

1. Full runnable code or full changes you made:
```
import numpy as np
import torch

from detectron2 import model_zoo
from detectron2.engine import DefaultPredictor
from detectron2.config import get_cfg
from detectron2.utils.testing import get_sample_coco_image
from detectron2.export.flatten import TracingAdapter

# --- LOAD MODEL ---

cfg = get_cfg()

# add project-specific config (e.g., TensorMask) here if you're not running a model in detectron2's core library
cfg.merge_from_file(model_zoo.get_config_file("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml"))
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5 # set threshold for this model

# Find a model from detectron2's model zoo. You can use the https://dl.fbaipublicfiles... url as well
cfg.MODEL.DEVICE = 'cpu'
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml")
predictor = DefaultPredictor(cfg)

# --- LOAD IMAGE ---
array = get_sample_coco_image(tensor=False)
tensor = get_sample_coco_image(tensor=True)
print(array.shape)
print(tensor.shape)

# --- DEFINE FUNCTIONS TO RUN CODE AS PREDICTOR, PURE PYTORCH MODEL AND TRACED MODEL

def process_image_with_predictor(image : np.ndarray):
""" Reimplements the preprocessing that the DefaultPredictor class executes in it's __call__ function before passing the
image to the model """
array = image
if predictor.input_format == "RGB":
# whether the model expects BGR inputs or RGB
array = array[:, :, ::-1]
height, width = array.shape[:2]
image = predictor.aug.get_transform(array).apply_image(array)
image = torch.as_tensor(image.astype("uint8").transpose(2, 0, 1))

inputs = {"image": image, "height": height, "width": width}
return inputs

def process_image_by_converting_to_tensor(image : np.ndarray):
height, width = image.shape[:2]
tensor = torch.as_tensor(image.astype("uint8").transpose(2, 0, 1))
inputs = {"image": tensor, "height": height, "width": width}
return inputs

def process_image(image : np.ndarray):
return process_image_by_converting_to_tensor(image)

def inference_with_traced_model(model, inputs):
if type(inputs) is dict:
image = inputs['image']
else:
image = inputs
predictions = model(image)
return predictions

def inference_with_model(model, inputs):
if type(inputs) is dict:
image = inputs['image']
else:
image = inputs
predictions = model([inputs])[0]
return predictions

# --- RUN MODEL: The Predictor and the model give the same result (if I apply preprocessing) ---

z = predictor(array)
predictor_pred_boxes = z['instances'].pred_boxes.tensor

model_boxes_same_as_predictor = inference_with_model(predictor.model, process_image_with_predictor(array))['instances'].pred_boxes.tensor
# Check if the tensors are all close
print(torch.allclose(model_boxes_same_as_predictor, predictor_pred_boxes))

# --- TRACING ---

def inference_func(model, image):
inputs = [{"image": image}]
return model.inference(inputs, do_postprocess=False)[0]

inputs = tuple([tensor])
wrapper = TracingAdapter(model, inputs, inference_func)
wrapper.eval()

with torch.no_grad():
trace_inputs = inputs
traced_model = torch.jit.trace(wrapper, trace_inputs)

# --- EVALUATE TRACED MODEL

model_boxes_without_processing = inference_func(predictor.model, tensor).get_fields()['pred_boxes'].tensor
# Since I don't run the preprocessing that is triggered in Predictor.__call__, the result is different to the predictor
print('Predictor Boxes Shape: ', predictor_pred_boxes.shape) # (12,4)
print('Model Boxes Shape: ', model_boxes_without_processing.shape) # (7,4)

# The traced model gives the same results as model_boxes_without_preprocessing (which makes sense)
traced_boxes_without_preprocessing = traced_model(tensor)[0]
print('Traced Boxes Shape: ', traced_boxes_without_preprocessing.shape)
print(torch.allclose(traced_boxes_without_preprocessing, model_boxes_without_processing)) # True

# --- PROBLEM ---
# After applying the same preprocessing, the traced model does not give the same result
traced_model_boxes_same_as_predictor = inference_with_traced_model(traced_model, process_image_with_predictor(array))[0]

print(torch.allclose(traced_model_boxes_same_as_predictor, predictor_pred_boxes)) # False
```
2. What exact command you run:
Paste this code into a python file and execute it

4. __Full logs__ or other relevant observations:
Running an image through the DefaultPredictor preprocessing that happens on using `__call__` and then passing the image through both the original model and the traced one gives different results

## Expected behavior:

I want to get a `.jit` representation of a pretrained Detectron model to push runtime performance.
Tracing the model works fine, but if I apply the preprocessing steps that the Detectron Predictor uses (I've reimplemented this in `process_image_with_predictor` function) before passing the data to the model, the traced model gives different results to the model it's tracing. Normally, I would expect that the traced model behaves as the original model.

## Environment:

```
------------------------------- ---------------------------------------------------------------------------------------
sys.platform darwin
Python 3.9.16 (main, Mar 8 2023, 04:29:24) [Clang 14.0.6 ]
numpy 1.25.2
detectron2 0.6 @/Users/USER/miniconda3/envs/py39/lib/python3.9/site-packages/detectron2
Compiler clang 14.0.3
CUDA compiler not available
DETECTRON2_ENV_MODULE
PyTorch 2.0.1 @/Users/USER/miniconda3/envs/py39/lib/python3.9/site-packages/torch
PyTorch debug build False
torch._C._GLIBCXX_USE_CXX11_ABI False
GPU available No: torch.cuda.is_available() == False
Pillow 10.0.0
torchvision 0.15.2 @/Users/USER/miniconda3/envs/py39/lib/python3.9/site-packages/torchvision
fvcore 0.1.5.post20221221
iopath 0.1.9
cv2 4.8.1
------------------------------- ---------------------------------------------------------------------------------------
PyTorch built with:
- GCC 4.2
- C++ Version: 201703
- clang 13.1.6
- LAPACK is enabled (usually provided by MKL)
- NNPACK is enabled
- CPU capability usage: NO AVX
- Build settings: BLAS_INFO=accelerate, BUILD_TYPE=Release, CXX_COMPILER=/Applications/Xcode_13.3.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -Wno-deprecated-declarations -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DUSE_PYTORCH_METAL_EXPORT -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DUSE_COREML_DELEGATE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=braced-scalar-init -Werror=range-loop-construct -Werror=bool-operation -Winconsistent-missing-override -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wvla-extension -Wno-range-loop-analysis -Wno-pass-failed -Wsuggest-override -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -Wconstant-conversion -Wno-invalid-partial-specialization -Wno-typedef-redefinition -Wno-unused-private-field -Wno-inconsistent-missing-override -Wno-constexpr-not-const -Wno-missing-braces -Wunused-lambda-capture -Wunused-local-typedef -Qunused-arguments -fcolor-diagnostics -fdiagnostics-color=always -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -DUSE_MPS -fno-objc-arc -Wno-unguarded-availability-new -Wno-unused-private-field -Wno-missing-braces -Wno-constexpr-not-const, LAPACK_INFO=accelerate, TORCH_DISABLE_GPU_ASSERTS=OFF, TORCH_VERSION=2.0.1, USE_CUDA=OFF, USE_CUDNN=OFF, USE_EIGEN_FOR_BLAS=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=OFF, USE_MKLDNN=OFF, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=OFF, USE_ROCM=OFF,
```

If your issue looks like an installation issue / environment issue,
please first check common issues in https://detectron2.readthedocs.io/tutorials/install.html#common-installation-issues

Start by running the complete Python reproduction and compare DefaultPredictor, predictor.model, and traced_model outputs. Read detectron2.export.flatten.TracingAdapter and the preprocessing represented by process_image_with_predictor; done means the traced model produces the same results as the original model after identical preprocessing.

Written by the indexing model from the issue text.