Frozen-bank fused-MoE forward returns corrupted bytes after unrelated GPU tests run (sglang fused-MoE combine-stage state bug)

Symptom

tests/models/test_glm52_fullparam_frozen_trunk_backward.py::test_cuda_frozen_bank_value_bytes_identical_with_and_without_grad_engagement fails in the GPU models CI shard (first surfaced on #82, which made sglang importable in the trainer env so this sglang-gated test runs in CI at all). It passes standalone and within its own file.

Minimal reproducer (H100, torch 2.11.0, sgl-kernel 0.4.5, triton 3.6.0)

Run in one process, in this order:

1. tests/models/test_batch_invariance_dense.py::test_dense_batch_composition_invariance_policy
2. tests/models/test_glm52_exact_routed_experts_qlora.py::test_routed_experts_literal_sampler_and_gradient_policy
3. the frozen-bank byte test

After steps 1–2, Glm52NativeBlockFP8Experts forward (_sglang_ep_native_routed_value → fused_experts_impl_strided → sglang fused_experts_impl, topk=1, EP-native local_ids with one -1 sentinel, routed_scaling_factor=1.5) returns structurally wrong and run-to-run nondeterministic outputs for bit-identical inputs: clean row norms [0.268, 0.536, 0.144, 0.287, 0.107] become [0.0, 0.0, 0.761, 0.761, 0.025] (± garbage up to 1e37 in some runs).

Evidence trail (all verified on H100)

Instrumented every Python-visible stage boundary, clean vs polluted process, identical fixture inputs:

• moe_align_block_size: inputs identical; sorted_ids/num_tokens_post_padded identical; expert_ids differs only in its uninitialized tail slot (block 5 of 5, beyond post-pad; torch.empty residue: 0x08080808 fresh page vs 0x3F800000 freed-float) — forcing that slot to -1 does not fix the failure.
• GEMM1 (invoke_fused_moe_kernel): input and output row norms bit-identical.
• Activation: intermediate_cache2 differs only in the filtered row (uninitialized by design).
• GEMM2: output (intermediate_cache3) row norms bit-identical.
• Final returned tensor: differs. The divergence enters between the last kernel invoke and the returned tensor (combine stage).

Ruled out empirically: torch backend flags (tf32/reduced-precision/deterministic — unchanged), xorl batch-invariant mode leak (off), bi_families_v2 selection (None), sglang exec-context (deterministic inference enabled in both), triton in-process JIT caches (cleared, no change), _down_moe_use_tma (forced off, no change), CUDA allocator state (empty_cache(), no change; NaN-priming a clean process does not reproduce), sgl_kernel.moe_sum_reduce primitive on fixed input (identical clean vs polluted).

Where this points

Persistent library state inside the pinned sglang fused-MoE stack's combine/epilogue path (sgl_kernel 0.4.5 + the vendored fused_moe.py orchestration), toggled by whatever the two polluter tests initialize. Needs debugging at the compiled-kernel level in xorl-sglang.

Interim state

The test is quarantined with xfail(strict=False) referencing this issue (PR #82). The frozen-bank byte contract itself holds when the process history is clean — verified 20/20 deterministic standalone runs.

Run the three named tests in one process and compare the frozen-bank byte test with a clean run. Trace Glm52NativeBlockFP8Experts through _sglang_ep_native_routed_value, fused_experts_impl_strided, the vendored fused_moe.py orchestration, and the sgl_kernel combine/epilogue path; done means the final tensor remains correct and deterministic after the two polluter tests.