mindspore-ai / mindspore-ai/hyper-parallel
【RFC】【WIP】hyper dcp 能力补齐（支持优化器、safetensors、plan缓存、异步保存等）
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 53
- Forks
- 63
- Avg merge
- 23h 45m
- Merged PRs (30d)
- 63
Description
按照基础能力和高阶能力划分，如下：
一、基础能力：
1、分布式保存、加载（包括权重、优化器等），每卡都写入分片ckpt
2、参数展平，针对优化器这种嵌套的字典，需要将其展平保存
3、支持pt和safetensors格式，safetensors为业界通用格式
4、去冗余保存，节点间存在冗余关系时，只落盘一份
5、负载均衡，保证每卡保存数据及量大体一致
6、支持集群和并行策略不变的断点续训能力
二、高阶能力：
1、saveplan缓存能力，整个训练过程中save plan只生成一次
2、异步保存，先同步到host内存，后台异步持久化到存储
3、load时在线resharding，在集群规模或者并行策略变化场景下，支持在线重分片和加载
4、文件偏移读取，在resharding场景下，只读取目标文件中的目标分片
5、参数广播能力，单卡读取，然后广播给冗余卡
6、支持stateful对象
schema_version: 1
source: gitcode
gitcode_repo: mindspore/hyper-parallel
gitcode_issue: 77
source_url: https://gitcode.com/mindspore/hyper-parallel/issues/77
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named. Start by locating the existing hyper DCP implementation and mapping it against the listed basic capabilities, then review the advanced requirements for save-plan caching, asynchronous persistence, resharing, offset reads, broadcasting, and stateful objects. Done is not yet defined beyond implementing this broad RFC scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100