Any recommendation for distillation PE？

Hi, thanks for your great work on PE!

I’ve tried it on my perception model and got good results.

Now I’m working on distilling a smaller ViT version and wonder:

1. Are there any recommended distillation strategies for small ViT models using only visual data? Would that be less effective than the CLIP-style distillation mentioned in Section C.3?

2. Is there any chance the distillation training code will be released?

Appreciate any guidance you can offer!

@shuminghu @dbolya @PeizeSun @mmaaz60 @Jiankai-Sun