alibaba / alibaba/ROLL

Missing ETP Support in mca_adapter Under Expert Parallelism

Open
#290 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3.4k
Forks
312
Avg merge
1h 2m
Merged PRs (30d)
2

Description

Could we consider adding support for Megatron-MoE's ETP (Expert-Tensor Parallel) sharding?
Right now, when initializing expert-parallel in MoE, it inherits the non-MoE tensor-parallel group by default.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the mca_adapter expert-parallel initialization described in the issue and trace how the non-MoE tensor-parallel group is selected. Determine the expected Megatron-MoE ETP sharding behavior and define tests that distinguish the ETP group from the inherited tensor-parallel group; done means ETP initialization uses the correct sharding.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.