Here are some questions about soft MoE

1. According to theory, image will transform to token(patch) first, and will become slots by weight then. I would like to know, in image aspect, where the program part will offer the segmentation of origin image, in order to let image become tokens. For example, if we have the image for 32x32, we set sequence length to 16 (meaning that we have 16 slots), and we set experts to 16 too. But the image just transform to slots directly, we don't see the transformation from image to tokens in program.
Shortly, tokens only depend on every pixel in original image, not depend on the patch segemented by original image.

2. I would like to know what loss function and optimizer you guys often to use with soft MoE, because we want to train some data (about 50000 images) with soft MoE in 4090*2.