Training panoptic segmentation model on COCO data fails

## Instructions To Reproduce the Issue:

I want to train the panoptic segmentation model on a custom data set, however, before I prepare my own custom data set I wanted to make sure that the training works on the COCO dataset. I saw @ppwwyyxx comment that training panoptic segmentation on a custom data set is not supported but I saw other people discussing about successfully doing that so I thought I would try that, too.

Since I don't need a large data set for this trial, I decided to train on COCO val2017 data set (rather than train2017). I downloaded these zip files:
http://images.cocodataset.org/zips/val2017.zip
http://images.cocodataset.org/annotations/panoptic_annotations_trainval2017.zip
http://images.cocodataset.org/annotations/stuff_annotations_trainval2017.zip

I am using detectron2.data.datasets.register_coco_panoptic_separated as suggested in [1691](https://github.com/facebookresearch/detectron2/issues/1691)

However, when I try to run the code below, I get an error "IndexError: Target 202 is out of bounds." from torch._C._nn.cross_entropy_loss.

As far as I understand, COCO data set contains 80 "thing" classes and 53 "stuff" classes. The category IDs in panoptic_val2017.json are from 1 to 200. So the number of targets should be either 80 + 53 + 1 = 134 or 200 at most. So where this "Target 202" come from???

Interestingly, if I set cfg.MODEL.SEM_SEG_HEAD.NUM_CLASSES = 203, the model trains successfully (not sure if the resulting model is meaningful though).

Another interesting point is that if I uncomment the line
sem_seg_root = "/scratch2/coco/from_website/panoptic2017/panoptic_annotations_trainval2017/annotations/panoptic_val2017/panoptic_val2017/"

Then even setting cfg.MODEL.SEM_SEG_HEAD.NUM_CLASSES = 250 does not help, because I get error "IndexError: Target 251 is out of bounds."

So since changing sem_seg_root changes the number of targets, I assume that the number of targets comes from semantic segmentation bitmaps. Is it equal to the number of different colors in semantic bitmaps? For panoptic masks the bitmap colors are mapped to "segment_info.id" field in panoptic_val2017.json using formula ids=R+G*256+B*256^2 (as instructed in https://cocodataset.org/#format-data), however, I'm not sure how semantic segmentation mask colors are mapped to category info in the json files.

A related question: does the custom panoptic segmentation training pipeline use panoptic bitmaps (panoptic_root) at all, or does it only use semantic bitmaps (sem_seg_root)?

1. Full runnable code or full changes you made:
```
from detectron2.data.datasets import register_coco_panoptic_separated

from detectron2 import model_zoo
from detectron2.engine import DefaultTrainer
from detectron2.config import get_cfg

image_root = "/scratch2/coco/from_website/val2017/"
panoptic_root = "/scratch2/coco/from_website/panoptic2017/panoptic_annotations_trainval2017/annotations/panoptic_val2017/panoptic_val2017/" # from http://images.cocodataset.org/annotations/panoptic_annotations_trainval2017.zip
panoptic_json = "/scratch2/coco/from_website/panoptic2017/panoptic_annotations_trainval2017/annotations/panoptic_val2017.json" # from http://images.cocodataset.org/annotations/panoptic_annotations_trainval2017.zip
sem_seg_root = "/scratch2/coco/from_website/stuff2017/stuff_annotations_trainval2017/annotations/stuff_val2017_pixelmaps/" # from http://images.cocodataset.org/annotations/stuff_annotations_trainval2017.zip
#sem_seg_root = "/scratch2/coco/from_website/panoptic2017/panoptic_annotations_trainval2017/annotations/panoptic_val2017/panoptic_val2017/"
instances_json = "/scratch2/coco/from_website/val2017/annotations/instances_val2017.json" # from http://images.cocodataset.org/zips/val2017.zip

register_coco_panoptic_separated("my_dataset", {}, image_root, panoptic_root, panoptic_json, sem_seg_root, instances_json)

cfg = get_cfg()
cfg.merge_from_file(model_zoo.get_config_file("COCO-PanopticSegmentation/panoptic_fpn_R_50_1x.yaml"))
cfg.MODEL.DEVICE='cpu'
cfg.DATASETS.TRAIN = ("my_dataset_separated",)

#cfg.MODEL.ROI_HEADS.NUM_CLASSES = 53
cfg.MODEL.SEM_SEG_HEAD.NUM_CLASSES = 200
#cfg.MODEL.SEM_SEG_HEAD.NUM_CLASSES = 203
#cfg.MODEL.SEM_SEG_HEAD.NUM_CLASSES = 250

cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-PanopticSegmentation/panoptic_fpn_R_50_1x.yaml")
cfg.SOLVER.IMS_PER_BATCH = 1 # This is the real "batch size" commonly known to deep learning people.

cfg.INPUT.MASK_FORMAT = "bitmask"

trainer = DefaultTrainer(cfg)
trainer.resume_or_load(resume=False)

trainer.train()
```
2. What exact command you run:

I saved the above commands in train_model_panoptic_issue.py and ran:
python train_model_panoptic_issue.py

3. __Full logs__ or other relevant observations:
```
[04/06 10:10:14 d2.engine.hooks]: Total training time: 0:00:03 (0:00:00 on hooks)
[04/06 10:10:14 d2.utils.events]: iter: 0 lr: N/A
Traceback (most recent call last):
File "train_model_panoptic_issue.py", line 38, in
trainer.train()
File "/usr/local/lib/python3.8/dist-packages/detectron2/engine/defaults.py", line 484, in train
super().train(self.start_iter, self.max_iter)
File "/usr/local/lib/python3.8/dist-packages/detectron2/engine/train_loop.py", line 149, in train
self.run_step()
File "/usr/local/lib/python3.8/dist-packages/detectron2/engine/defaults.py", line 494, in run_step
self._trainer.run_step()
File "/usr/local/lib/python3.8/dist-packages/detectron2/engine/train_loop.py", line 274, in run_step
loss_dict = self.model(data)
File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/detectron2/modeling/meta_arch/panoptic_fpn.py", line 127, in forward
sem_seg_results, sem_seg_losses = self.sem_seg_head(features, gt_sem_seg)
File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/detectron2/modeling/meta_arch/semantic_seg.py", line 239, in forward
return None, self.losses(x, targets)
File "/usr/local/lib/python3.8/dist-packages/detectron2/modeling/meta_arch/semantic_seg.py", line 263, in losses
loss = F.cross_entropy(
File "/usr/local/lib/python3.8/dist-packages/torch/nn/functional.py", line 2996, in cross_entropy
return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
IndexError: Target 202 is out of bounds.
```

## Expected behavior:

The expected behavior is that model training works when the number of segmentation heads is set to num_thing_classes + num_stuff_classes + 1:
cfg.MODEL.SEM_SEG_HEAD.NUM_CLASSES = 80 + 53 + 1

## Environment:

Paste the output of the following command:
```
2023-04-06 10:16:41 URL:https://raw.githubusercontent.com/facebookresearch/detectron2/main/detectron2/utils/collect_env.py [8525/8525] -> "collect_env.py" [1]
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
--------------------- ----------------------------------------------------------------
sys.platform linux
Python 3.8.10 (default, Nov 14 2022, 12:59:47) [GCC 9.4.0]
numpy 1.22.3
detectron2 0.6 @/usr/local/lib/python3.8/dist-packages/detectron2
Compiler GCC 9.4
CUDA compiler not available
DETECTRON2_ENV_MODULE
PyTorch 1.11.0+cu102 @/usr/local/lib/python3.8/dist-packages/torch
PyTorch debug build False
GPU available No: torch.cuda.is_available() == False
Pillow 9.3.0
torchvision 0.12.0+cu102 @/usr/local/lib/python3.8/dist-packages/torchvision
fvcore 0.1.5.post20221213
iopath 0.1.9
cv2 4.5.4
--------------------- ----------------------------------------------------------------
PyTorch built with:
- GCC 7.3
- C++ Version: 201402
- Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
- Intel(R) MKL-DNN v2.5.2 (Git Hash a9302535553c73243c632ad3c4c80beec3d19a1e)
- OpenMP 201511 (a.k.a. OpenMP 4.5)
- LAPACK is enabled (usually provided by MKL)
- NNPACK is enabled
- CPU capability usage: NO AVX
- Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=10.2, CUDNN_VERSION=7.6.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.11.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=OFF, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF,
```