GroupedBatchSampler in detection reference

🐛 Describe the bug

Since we have multiscale and lsj augmentations which use fix sizes, we no longer resize the images in the transforms of GeneralizedRCNNTransform. However, we still use the GroupedBatchSampler but for me the grouping by aspect ratio is no longer mandatory. Am I right ?

Even though I don't think it is very useful (if the statement above is true), we can easily patch it.

We can change in the training reference script the following lines:
https://github.com/pytorch/vision/blob/bd19fb8ea9b1f67df2a2a1ee116874609ad3ee8c/references/detection/train.py#L191-L195
as

    if args.aspect_ratio_group_factor >= 0 and args.data_augmentation not in ["multiscale", "lsj"]:
        group_ids = create_aspect_ratio_groups(dataset, k=args.aspect_ratio_group_factor)
        train_batch_sampler = GroupedBatchSampler(train_sampler, group_ids, args.batch_size)
    else:
        train_batch_sampler = torch.utils.data.BatchSampler(train_sampler, args.batch_size, drop_last=True)

Versions

[pip3] numpy==1.21.5
[pip3] torch==1.11.0
[pip3] torchvision==0.12.0
[conda] blas 1.0 mkl
[conda] cudatoolkit 10.2.89 hfd86e86_1
[conda] ffmpeg 4.3 hf484d3e_0 pytorch
[conda] mkl 2021.4.0 h06a4308_640
[conda] mkl-service 2.4.0 py37h7f8727e_0
[conda] mkl_fft 1.3.1 py37hd3c417c_0
[conda] mkl_random 1.2.2 py37h51133e4_0
[conda] numpy 1.21.5 py37he7a7128_1
[conda] numpy-base 1.21.5 py37hf524024_1
[conda] pytorch 1.11.0 py3.7_cuda10.2_cudnn7.6.5_0 pytorch
[conda] pytorch-mutex 1.0 cuda pytorch
[conda] torchvision 0.12.0 py37_cu102 pytorch

Inspect references/detection/train.py around lines 191-195 and compare the existing GroupedBatchSampler path with the proposed BatchSampler path for multiscale and lsj augmentation. Confirm whether aspect-ratio grouping is still needed, then verify that the detection training reference uses the appropriate sampler for each augmentation mode.