MFSDP v2: overlap DP-inner and DP-outer communication
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 4.5k
- Avg merge
- 4d 6h
- Merged PRs (30d)
- 271
Description
Follow-up to #5616: explore overlapping DP-inner and DP-outer communication in MFSDP v2.
The two communications use different network domains, and early experience suggests that overlapping them could reduce latency significantly. The implementation should follow the runtime scheduling model in `megatron/core/distributed/fsdp/src/docs/runtime_schedule.md`.
We should first measure the benefit and use those results to guide the implementation and scheduling design.
Contributor guide
Research direction
Read megatron/core/distributed/fsdp/src/docs/runtime_schedule.md and review the MFSDP v2 communication paths described in follow-up issue #5616. First measure whether DP-inner and DP-outer communication overlap reduces latency, then use those results to define the scheduling design and implementation scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100