Size mismatch error between images and masks AFTER loading dataset

## Instructions To Reproduce the Issue:
Trying to train a model using panoptic_fpn

torch.cuda.empty_cache()
config_file = "COCO-PanopticSegmentation/panoptic_fpn_R_50_3x.yaml"
cfg = get_cfg()
cfg.MODEL.DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
cfg.merge_from_file(model_zoo.get_config_file(config_file))
cfg.DATASETS.TRAIN = (f"{data_name}_separated",)
cfg.DATASETS.TEST = ()
cfg.DATALOADER.NUM_WORKERS = 0
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url(config_file)
cfg.SOLVER.IMS_PER_BATCH = 1
cfg.SOLVER.BASE_LR = 0.00025
cfg.SOLVER.MAX_ITER = 5000
cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 16
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 1
cfg.MODEL.SEM_SEG_HEAD.NUM_CLASSES = 2
cfg.MODEL.PANOPTIC_FPN.NUM_CLASSES = 1
cfg.SOLVER.AMP.ENABLED = True

MetadataCatalog.get(cfg.DATASETS.TRAIN[0]).set(thing_classes=things, stuff_classes=stuff, thing_dataset_id_to_contiguous_id={1: 0})

Example images that are causing error
[bad_imgs_2.zip](https://github.com/user-attachments/files/16630840/bad_imgs_2.zip)

## Expected behavior:
I've run this exact code with a different custom image set and the model ran fine.
There are a few images in this image set (i.e. example images and masks below attached) that we have found to through this size error that says the masks and images are different sizes. I manually checked the dimensions of the masks and images using img properties on my PC, and it says the images and masks are the same dimensions. However, when I check the image and mask sizes after the model loads the dataset, the sizes are different.
My running theory is that there is some transformation that occurs with the dataset mapper that changes the dimensions of the masks, but I'm not sure how to see if that is the case or if something else in the code is changing the dimensions of the mask and the image separately.

Any and all help trying to solve this error would be appreciated.

**Example images used in dataset in this repo: ([(https://github.com/zacklew/bad_images)])*

## Environment:

sys.platform linux
Python 3.12.4 | packaged by Anaconda, Inc. | (main, Jun 18 2024, 15:12:24) [GCC 11.2.0]
numpy 1.26.4
detectron2 0.6 @/home/computational/anaconda3/lib/python3.12/site-packages/detectron2
Compiler GCC 11.2
CUDA compiler CUDA 12.5
detectron2 arch flags 8.6
DETECTRON2_ENV_MODULE
PyTorch 2.3.1+cu121 @/home/computational/anaconda3/lib/python3.12/site-packages/torch
PyTorch debug build False
torch._C._GLIBCXX_USE_CXX11_ABI False
GPU available Yes
GPU 0 NVIDIA RTX A2000 12GB (arch=8.6)
Driver version 555.58.02
CUDA_HOME /home/computational/anaconda3
Pillow 10.3.0
torchvision 0.18.1+cu121 @/home/computational/anaconda3/lib/python3.12/site-packages/torchvision
torchvision arch flags 5.0, 6.0, 7.0, 7.5, 8.0, 8.6, 9.0
fvcore 0.1.5.post20221221
iopath 0.1.9
cv2 4.10.0
------------------------------- ------------------------------------------------------------------------------------
PyTorch built with:
- GCC 9.3
- C++ Version: 201703
- Intel(R) oneAPI Math Kernel Library Version 2023.1-Product Build 20230303 for Intel(R) 64 architecture applications
- Intel(R) MKL-DNN v3.3.6 (Git Hash 86e6af5974177e513fd3fee58425e1063e7f1361)
- OpenMP 201511 (a.k.a. OpenMP 4.5)
- LAPACK is enabled (usually provided by MKL)
- NNPACK is enabled
- CPU capability usage: AVX512
- CUDA Runtime 12.1
- NVCC architecture flags: -gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90
- CuDNN 8.9.2
- Magma 2.6.1
- Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=8.9.2, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.3.1, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=1, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF,

Start by reproducing the failure with COCO-PanopticSegmentation/panoptic_fpn_R_50_3x.yaml and the images in bad_imgs_2.zip or the linked bad_images repository. Inspect the dataset mapper and compare image and mask dimensions before and after loading and transformations. Done means identifying where the mismatch is introduced and documenting a confirmed fix or required dataset change.

Written by the indexing model from the issue text.