NVIDIA-NeMo / NVIDIA-NeMo/Automodel
Merge moe and fsdp paths
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 963
- Forks
- 318
- Avg merge
- 3d 20h
- Merged PRs (30d)
- 143
Description
Right now these two are separate paths. For example, if a user wants to use EP then a paralllelize_fn is used which will then apply FSDP itself, but if a user just wants FSDP2 then the FSDP2Manager is used. This incurs confusion and we should clean this up s.t. parallelisms are applied via the managers (FSDP2Manager, MegatronFSDPManager, DDPManager).
Additionally, we need to merge the moe and device meshes into one.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing how parallelize_fn, FSDP2Manager, MegatronFSDPManager, and DDPManager currently apply parallelisms. Compare the MoE and device-mesh handling in these paths. Done means parallelisms are applied through the managers and the MoE and device meshes are merged without preserving the confusing separate paths.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100