MoE exploration for OELLM

💡 How do we decide the active and total parameter size (i.e., sparsity) for the sparse "baby" or "flagship" model that we want to train?

🧠 The "Why" (Problem & Motivation)

In literature, we see that with increasing total parameters, making the models increasingly sparse is better. But how far can we push it? Also is there an optimal sparsity depending on the token budget that we want to train on?

🚀 The Big Idea

We want to design a grid over HPs and sparsity to answer the above question, and also make recommendations for the OpenEuroLLM sparse model dimensions. We also want to develop a recipe to train "healthy" MoE models that transfers across scales.

🎯 The Dream Outcome

• MoEs better than the dense models
• Proper expert utilization and specialization (metrics being developed and tracked for this)

---

⚙️ Prototype (Optional)

Early results on English data resulted in poor MoEs! We observed router collapse and dead experts, see more in this dashboard!

❓ Things I'm Figuring Out (Optional)

What's still blurry? What assumptions are we making or need to research?

📎 References and Similar Works

Add the references you used while pitching the idea

---

💬 Discussion Notes

Drop the TL;DR of decisions, pivots, and feedback here as people weigh in.

🏷️ Reminder: Don't forget to pop the appropriate label on this before you publish on board!

Start by reviewing the early English-data results in the linked MoE analysis dashboard, focusing on router collapse and dead experts. Done means producing a hyperparameter and sparsity grid, recommendations for sparse model dimensions, and a transferable training recipe supported by expert utilization and specialization metrics.