mindspore-ai / mindspore-ai/hyper-parallel

Fix stale strategy state and trainer resource lifecycle

Open
#892 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
53
Forks
63
Avg merge
23h 45m
Merged PRs (30d)
63

Description

问题背景

本 issue 用于跟踪 GitCode 社区问题 org-issues#43396 在 HyperParallel 仓内的修复。

问题范围

  1. ND 多模态策略搜索更新候选并行配置时,embedding 和 recompute 分片因继续读取解析阶段配置而产生陈旧的显存估计。
  2. BaseTrainer.train 异常退出时未统一停止后台数据预取、结束 callback、释放 profiler 并销毁分布式进程组。
  3. Checkpoint Registry 的注册项无法通过 MutableMapping.popclear 完整移除,且 registry 名称列表在实例间共享。
  4. Callback 强引用 Trainer 形成引用环,使 Trainer 和相关显存的释放依赖 cyclic GC。

预期结果

  • 策略搜索中的所有子配置按当前候选并行度重新计算分片因子。
  • 训练正常结束或异常退出时均执行完整、可重复的资源清理。
  • Registry 符合 MutableMapping 的删除语义,实例状态互不污染。
  • Callback 不延长 Trainer 生命周期。
  • 为上述行为补充硬件无关的单元测试。

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at BaseTrainer.train, the ND multimodal strategy-search configuration update, the Checkpoint Registry, and callback ownership; trace how each currently retains state or resources. Add hardware-independent unit tests covering candidate-based shard recalculation, normal and exceptional cleanup, complete registry deletion and instance isolation, and callback lifetime; done means all four behaviors are verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, testing-qa
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.