[bug] DeviceMesh 在 ms.DeviceCtx("meta") 上下文内构造时崩溃

该问题是怎么引起的？

hyper_parallel 两处实现没考虑 PyNative 延迟初始化(ms.DeviceCtx("meta") 包裹 fully_shard)+ TP/EP + FSDP 端到端场景:

1. DeviceMesh 在 meta context 内构造时 metadata tensor 落 meta:_convert_rank_map_to_tensor / _refresh_mesh_view 内部 Tensor(...) 被 pin 到当前 ambient device。DeviceMesh.concatenate 在 fully_shard 的 _get_base_spmd_placements 内被调用时 metadata 落 meta,后续
   asnumpy() / nonzero() 在 Ascend 上失败或 SIGSEGV。仅 TP/EP shard DTensor 参数时触发。

2. HCCL 拒绝非 contig grad:_to_local_unsharded_grad 是所有 reduce 路径的统一入口,其底层 _normalize_unsharded_grad_to_local 可能返回 view(DTensor.to_local() / redistribute())。DistCommAllReduce / DistCommReduceScatter 拒收非 contig tensor。错误异步触发(wait_pipeline
   在 post_backward 暴露),源头难定位。EP+FSDP 拓扑更易踩到。

重现步骤

msrun --worker_num=8 --local_worker_num=8 --master_port=8128 \
      --log_dir=logs --join=True \
      run_mindformer.py --config cfg.yaml --mode 1

┌──────────────────────────┬─────────────────┐
│           配置           │      触发       │
├──────────────────────────┼─────────────────┤
│ TP=2 / TP=2+EP=2 + FSDP  │ 问题 1          │
├──────────────────────────┼─────────────────┤
│ TP=1 + EP=2 + FSDP       │ 问题 1 + 问题 2 │
├──────────────────────────┼─────────────────┤
│ TP=1 EP=1(单卡 baseline) │ 都不触发        │
└──────────────────────────┴─────────────────┘

报错信息

问题 1:

File "hyper_parallel/core/fully_shard/hsdp_param.py:200", in _get_base_spmd_placements
    self._spmd_mesh = DeviceMesh.concatenate([self.mesh_info.mesh, self._orig_dtensor_mesh])
File "hyper_parallel/core/dtensor/device_mesh.py:238", in _refresh_mesh_view
    full_mesh_np = self._layout.remap_to_numpy(platform.tensor_to_numpy(self._rank_map))
RuntimeError: Not support copy between src:meta and dst:CPU

绕过该点后,(meta_tensor == int).nonzero() SIGSEGV(worker exit code -11,无 Python traceback)。

问题 2:

File "hyper_parallel/platform/mindspore/fully_shard/state.py:513", in post_backward
    if not hsdp_param.sharded_param.requires_grad:
File "mindspore/common/parameter.py:438", in param_info
    Tensor_.wait_pipeline(self)
ValueError: DistCommAllReduce does not support not-contiguous tensor.
Please call tensor.contiguous() firstly.

注意 Python 栈是 wait_pipeline 等待时暴露,不是真正发起 AllReduce 的栈。


<!-- bot3-issue-meta -->
schema_version: 1
source: gitcode
gitcode_repo: mindspore/hyper-parallel
gitcode_issue: 134
source_url: https://gitcode.com/mindspore/hyper-parallel/issues/134
<!-- /bot3-issue-meta -->

Start with hyper_parallel/core/dtensor/device_mesh.py, especially _convert_rank_map_to_tensor, _refresh_mesh_view, and DeviceMesh.concatenate, then reproduce the listed TP/EP and FSDP configurations. Trace the reduce paths through hyper_parallel/platform/mindspore/fully_shard/state.py and _to_local_unsharded_grad. Done means the meta-context mesh path and the affected gradient reductions complete without the reported crashes or non-contiguous-tensor error.