modelscope / modelscope/ms-swift
能否支持megatron hsdp
Open
Nobody has claimed this yet.
enhancement
stale
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 1.7k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 136
Description
Checklist / 检查清单
- I have searched existing issues, and this is a new feature request. / 我已经搜索过现有的 issues，确认这是一个新的 Feature Request。
Feature Request Description / Feature Request 描述
我之前训练32B的qwen3，发现megatron开tp训练速度比deepspeed zero3慢。希望megatron支持hsdp，这样我就可以all in megatron了。
Pull Request / Pull Request 信息
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The request names Megatron training, HSDP, and DeepSpeed ZeRO-3 but no files or tests. Start by locating the existing Megatron and ZeRO-3 integration points, then compare their parallelism paths and define the expected HSDP behavior. Done requires an agreed implementation scope and validation against the reported 32B Qwen3 training case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100