Is it possible to customize an MOE model configuration in the ablations directory and train it?

Hello,

I am currently working with this repository and I am interested in customizing a mixture-of-experts (MOE) model configuration within the `ablations `directory. Specifically, I would like to modify certain aspects of the model and training parameters to create a customized MOE model for training.

Could you please confirm if it is possible to create and train a custom MOE configuration in the `ablations `directory? If so, I would appreciate any guidance or suggestions on how to modify the configuration and proceed with the training process.

Thank you in advance for your help!

Best regards,
Rongrong Zhou