Missing ETP Support in mca_adapter Under Expert Parallelism
Open
- Dominant language
- Python
- Stars
- 3.4k
- Forks
- 312
- Avg merge
- 1h 2m
- Merged PRs (30d)
- 2
Description
Could we consider adding support for Megatron-MoE's ETP (Expert-Tensor Parallel) sharding?
Right now, when initializing expert-parallel in MoE, it inherits the non-MoE tensor-parallel group by default.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the mca_adapter expert-parallel initialization described in the issue and trace how the non-MoE tensor-parallel group is selected. Determine the expected Megatron-MoE ETP sharding behavior and define tests that distinguish the ETP group from the inherited tensor-parallel group; done means ETP initialization uses the correct sharding.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100