scale input image size problem

Hi, I meet a problem when try to scale input image size, 1024 is to big so I scale it to 224, but if I use the "use_rel_pos=True", the input image_size args in all ckpt is 1024, it will report error (size mismatch at blocks 2, 5、8、11) at Attention() module in VIT.

so, is it feasible to set encoder_global_attn_indexes = [] to avoid this error? Will the performance degrade？

Start with the Attention() module and the checkpoint image_size setting described in the report, focusing on the size mismatch at blocks 2, 5, 8, and 11. Determine whether a 224-sized input can work with use_rel_pos=True and whether setting encoder_global_attn_indexes to [] changes the reported behavior or performance.

Written by the indexing model from the issue text.