Support for Facebook's new SeamlessM4T (Multilingual + Multimodal)

Facebook just released a new multimodal model for multiple languages. I would assume it's the successor to NLLB. One model to rule them all. It would be amazing to have CT2 support for this to further reduce the size of the large model. If I remember correctly, when I used Whisper large and NLLB-200 medium, I was using about 9-10 GB of VRAM with what should be under 3B parameters. Switching to CT2s whisper large-v2 and NLLB-200 medium (both float16) took me to 5-6 GB of VRAM. I'm hoping that with CT2 support for SeamlessM4T we can see similar improvements with negligible loss of accuracy all while maintaining solid multimodal metrics. That being said, in the future if there is support for SM4T, would you be as so kind as to include metrics of vanilla SM4T and CT2's SM4T for as many tasks (e.g. S2TT, T2TT, etc.) possible? If not, maybe a script so we can analyze it ourselves?

Thanks, hopefully it's not much of an ask to add support for in the future and that other people can take advantage of this.

Website: https://ai.meta.com/resources/models-and-libraries/seamless-communication/
Code: https://github.com/facebookresearch/seamless_communication
Paper: https://ai.meta.com/research/publications/seamless-m4t/
Blog Post: https://ai.meta.com/blog/seamless-m4t/

The issue names no CTranslate2 files, tests, or entry points. Start by reviewing the linked seamless_communication code and the CTranslate2 model-support entry points; done would include SeamlessM4T support plus a way to compare vanilla and CTranslate2 metrics across the requested tasks.