mindspore-ai / mindspore-ai/hyper-parallel
refactor: 测试端口硬编码改为原子化自动分配
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 53
- Forks
- 63
- Avg merge
- 23h 45m
- Merged PRs (30d)
- 63
Description
背景
44 个 ST shell 文件（tests/torch/shard/ops/test_parallel_op_*.py）中硬编码了
master_port（范围 10359–10999），在 CI 门禁使用 pytest-xdist 并行执行时会发
生端口冲突。每次新增测试需手动分配不冲突的端口，维护成本高且不可靠。
目标
消除所有硬编码端口，实现原子化自动分配：
allocate_port()— 基于fcntl.flock(LOCK_EX)的原子文件锁计数器，保证并发
进程永不拿到相同端口torchrun_case()—master_port参数默认None，自动分配TorchCase—master_port参数默认None，parallel_run()在父进程预分配
端口后 spawn 子进程
端口分配机制
| 特性 | 说明 |
|---|---|
| 原子性 | fcntl.flock(LOCK_EX) 内核级排他锁，并发无竞争 |
| 循环范围 | 40000–59999（counter % 20000），溢出自动回绕 |
| 文件路径 | /tmp/hp_port_counter_<uid>，不同用户隔离 |
| 预分配 | parallel_run() 在父进程分配端口后再 fork，子进程无竞争 |
schema_version: 1
source: gitcode
gitcode_repo: mindspore/hyper-parallel
gitcode_issue: 158
source_url: https://gitcode.com/mindspore/hyper-parallel/issues/158
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the 44 files matching tests/torch/shard/ops/test_parallel_op_*.py and trace their current master_port usage into torchrun_case() and TorchCase. Verify how parallel_run() spawns children, then implement the specified allocation and run the affected parallel tests under pytest-xdist. Done means hard-coded ports are removed and concurrent runs receive distinct ports.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100