Qwen3.5-35B-A3B at EP1 with CP>1 segfaults in Transformer Engine's grouped GEMM on real-data routing (likely empty local experts)

Found by the expanded cost-model calibration campaign on Qwen3.5-35B-A3B (bf16, active LoRA slot, dev/trainer_rank_landing_acceptance.py --phase cost-calibrate), 4x H200.

Symptom. With expert parallelism forced to 1 (all 256 experts replicated on every rank) and context parallelism > 1, the first forward on real Ellavox histories segfaults on some ranks inside Transformer Engine's grouped GEMM:

!!!!!!! Segfault encountered !!!!!!!
  nvte_multi_tensor_gemm
  transformer_engine::pytorch::te_general_grouped_gemm(...)
  ... (dynamo frames) ...
exitcode -11 (SIGSEGV) on rank 1 (CP2) / ranks 1 and 3 (CP4)

Pattern over 40 cells:

So the crash needs (a) the EP1 replicated-expert dispatcher path (TE grouped GEMM over all local experts), (b) few tokens per rank (CP shards a 12k-token group to ~6k / ~3k tokens per rank) and (c) real routing distributions; random-token synthetic cells with the same shapes never crash. The consistent explanation is local experts that receive zero routed rows: with 256 local experts and a few thousand skewed tokens per rank, some experts end up with m = 0 groups, which the TE grouped GEMM call does not tolerate. HybridEP shapes (EP2/EP4, 64–128 local experts, more tokens per expert) never hit it.

Note the runtime's default topology sets EP to the visible GPU count for MoE models, so this configuration only arises when EP is pinned below the world size (as the calibration lattice does to separate CP from EP effects). It should still fail with a clear error, or handle empty expert groups, rather than SIGSEGV.

Repro: sky launch dev/trainer_rank_cost_calibration_lattice.sky.yaml --env MODEL=Qwen/Qwen3.5-35B-A3B --env SHAPES="1,2,1" --env CELLS="cal-ellavox:0:2" --env REPEAT=1 --env RUN_ID=<id> (Ellavox corpus mounted; PYTHONFAULTHANDLER=1 is set by the recipe). Logs with the native stack: scratch/trainer_rank_cost_calibration/l35-diag-20260904-0241/tp1-cp2-ep1-etp1-cal-ellavox-0-g2.log (local copies with the campaign artifacts).

For the calibration campaign the CP4/EP1 Ellavox cells of this class are recorded as unmeasurable; the CP-versus-EP separation on this model rests on the synthetic EP1 cells (all measured) plus the EP2/EP4 Ellavox cells.

Start with dev/trainer_rank_landing_acceptance.py and the sky calibration repro, then inspect the EP1 dispatcher path around Transformer Engine grouped GEMM. Compare the CP2/EP1 and CP4/EP1 Ellavox runs with the working EP2/EP4 cases, using the native stack in scratch/trainer_rank_cost_calibration/l35-diag-20260904-0241/tp1-cp2-ep1-etp1-cal-ellavox-0-g2.log. Done means the empty-local-expert case no longer segfaults and the affected calibration cells complete or fail with a clear error.