mindspore-ai / mindspore-ai/hyper-parallel
Fix stale strategy state and trainer resource lifecycle
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 53
- Forks
- 63
- Avg merge
- 23h 45m
- Merged PRs (30d)
- 63
Description
问题背景
本 issue 用于跟踪 GitCode 社区问题 org-issues#43396 在 HyperParallel 仓内的修复。
问题范围
- ND 多模态策略搜索更新候选并行配置时,embedding 和 recompute 分片因继续读取解析阶段配置而产生陈旧的显存估计。
BaseTrainer.train异常退出时未统一停止后台数据预取、结束 callback、释放 profiler 并销毁分布式进程组。- Checkpoint
Registry的注册项无法通过MutableMapping.pop或clear完整移除,且 registry 名称列表在实例间共享。 - Callback 强引用 Trainer 形成引用环,使 Trainer 和相关显存的释放依赖 cyclic GC。
预期结果
- 策略搜索中的所有子配置按当前候选并行度重新计算分片因子。
- 训练正常结束或异常退出时均执行完整、可重复的资源清理。
- Registry 符合
MutableMapping的删除语义,实例状态互不污染。 - Callback 不延长 Trainer 生命周期。
- 为上述行为补充硬件无关的单元测试。
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at BaseTrainer.train, the ND multimodal strategy-search configuration update, the Checkpoint Registry, and callback ownership; trace how each currently retains state or resources. Add hardware-independent unit tests covering candidate-based shard recalculation, normal and exceptional cleanup, complete registry deletion and instance isolation, and callback lifetime; done means all four behaviors are verified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, testing-qa
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100