intelligent-machine-learning / intelligent-machine-learning/dlrover

Add balance loss in atorch moe example

Open
#1,300 0 comments 0 reactions 0 assignees View on GitHub
Hacktoberfest todo
Dominant language
Python
Stars
1.7k
Forks
219
Avg merge
10h 11m
Merged PRs (30d)
11

Description

Add balance loss in [moe example](https://github.com/intelligent-machine-learning/dlrover/tree/master/atorch/examples/moe).

We add use aux loss to router in moe example, similar to aux loss implementation in[ megatron moe](https://github.com/NVIDIA/Megatron-LM/blob/0d89fc4c0d4394f915fffff11212d6957652337f/megatron/core/transformer/moe/router.py#L142
).

Contributor guide

Open the contributing guide

Research direction

Start in atorch/examples/moe and inspect how the router currently handles expert selection. Compare it with the linked Megatron MoE router implementation, focusing on its auxiliary balance-loss behavior. Done means the example uses an auxiliary loss in the router to improve expert load balance.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.