[Bug][MCore dev 0e6ac576] Intermittent CUDA runtime failures in DeepSeek-V4-0731 fused CSA/DSA with packed THD + CP32 on SM90

**Describe the bug**

We observed two low-frequency, non-deterministic CUDA runtime failures while
fine-tuning DeepSeek-V4-Flash with fused hybrid CSA/DSA, packed THD inputs,
and context parallelism 32 on SM90 GPUs.

The failures occurred after hundreds of successful iterations, on different
workers and in different fused-attention code paths:

1. `cudaErrorInvalidValue` surfaced from the cuDNN Frontend SM90 sparse-attention
backward path.
2. `cudaErrorUnknown` surfaced from the MCore packed-THD
`build_attention_indices` path.

Both ultimately passed through CUTLASS DSL / TVM-FFI. We do not know whether
they have the same root cause. They are grouped here because they occur under
the same fused CSA/DSA configuration and both overlap with compile-cache paths.
We are happy to split them if preferred.

`CUDA_LAUNCH_BLOCKING` was not enabled, so the reported call could also be the
first synchronization point rather than the kernel that originally caused the
error.

### Environment

- Megatron-Bridge:
[`af17edf52c514c58c79cd291b04a9b42bc5c57f3`](https://github.com/NVIDIA-NeMo/Megatron-Bridge/commit/af17edf52c514c58c79cd291b04a9b42bc5c57f3)
- Megatron Core dev:
[`0e6ac576fba8995fca541ceb6af5f35eb0e4f16e`](https://github.com/NVIDIA/Megatron-LM/commit/0e6ac576fba8995fca541ceb6af5f35eb0e4f16e)
- Python 3.12
- PyTorch 2.11.0+cu129
- CUDA driver 580.105.08
- Hopper-class SM90 GPUs, 32 nodes × 8 GPUs
- nvidia-cudnn-frontend 1.26.0
- nvidia-cudnn-cu12 9.16.0.29
- nvidia-cutlass-dsl 4.5.2
- apache-tvm-ffi 0.1.11
- cuda-python 12.9.0 / cuda-bindings 12.9.4
- Transformer Engine 2.14.1
- FlashMLA image-build pin: `b7643bd54521f563b839b98289b5cd048c062ba2`

A downstream MCore patch was applied. Its only CSA/DSA-adjacent change removes
`@torch.compile` from `csa_cp_utils._build_cp_indexer_layout`; it does not modify
the failing functions in `dsa_kernels.py`, `csa_cp_layout_kernels.py`, `csa.py`,
or `deepseek_v4_hybrid_attention.py`.

Our integration explicitly overrides the Bridge provider to set
`apply_dsa_kernel_fusion=True` on SM90. This deviation is disclosed because
older Bridge defaults disabled fused DSA on Hopper.

All 32 nodes had identical driver and package versions and identical hashes for
the relevant MCore and cuDNN Frontend source files.

### Relevant configuration

- BF16 parameters
- 43 layers, hidden size 4096
- 64 attention heads, one KV group, head dimension 512
- DSA indexer: 64 heads, head dimension 128, top-k 512
- `apply_dsa_kernel_fusion=True`
- `dsa_indexer_loss_coeff=0.0`
- `dsa_indexer_use_sparse_loss=True`
- TP=1, PP=8, CP=32, EP=32
- Packed THD inputs
- `sequence_packing_scheduler=dp_balanced`
- `variable_seq_lengths=True`
- `cp_partition_mode=contiguous`
- Global batch size 32
- Maximum sequence/packed-token budget: 262144
- Maximum post-CP local token budget: 8192
- Full activation recomputation, uniform, one layer per recompute unit
- CUDA graphs disabled (`cuda_graph_impl="none"`)

**Steps/Code to reproduce bug**

We do not yet have a small deterministic reproducer. Run the configuration
above for several hundred steps with dynamically packed, variable-length THD
inputs.

### Incident A: DSA sparse-attention backward

The run completed 395 steps and failed at step 396. The batch was partitioned
into seven dynamic microbatches, with these per-sample total lengths:

- mean: 49,536.47
- minimum: 25,834
- maximum: 114,663

Sanitized traceback:

```text
torch.autograd.backward(...)
-> megatron/core/transformer/experimental_attention_variant/dsa_kernels.py:377
DSA backward
-> cudnn.DSA.sparse_attention_backward_wrapper(...)
-> cudnn/deepseek_sparse_attention/sparse_attention_backward/api.py
-> flash_attn_bwd_sm90(...)
-> sparse_attention_backward/_interface_sm90.py:295
flash_attn_bwd_sm90.compile_cache[compile_key](...)
-> nvidia_cutlass_dsl/.../tvm_ffi_provider.py:588

RuntimeError: CUDA Error: cudaErrorInvalidValue
```

An earlier independent run used the same model/configuration and identical
sample order and microbatch packing for step 396. It completed the step
successfully and continued for another 49 steps.

### Incident B: packed-THD CSA layout forward

A separate run completed 566 steps and failed at step 567. The batch was
partitioned into ten dynamic microbatches, with these lengths:

- mean: 65,273.09
- minimum: 25,193
- maximum: 164,544

Sanitized traceback:

```text
megatron/core/transformer/experimental_attention_variant/
deepseek_v4_hybrid_attention.py:315
-> csa.py:1860, _forward_thd_cp
-> csa.py:2483
csa_cp_layout_kernels.build_attention_indices(...)
-> csa_cp_layout_kernels.py:811
build_attention_indices
-> csa_cp_layout_kernels.py:641
_run_compiled_launch
-> compiled(...)
-> nvidia_cutlass_dsl/.../tvm_ffi_provider.py:588

RuntimeError: CUDA Error: cudaErrorUnknown
```

The next retry processed exactly the same step-567 sample order, microbatch
packing, and length statistics successfully. The worker that previously
reported the failure was still participating with all eight GPUs.

### Additional observations

- No CUDA OOM preceded either failure.
- No NCCL/RDMA error was the first reported error.
- Kernel-log inspection found no NVRM Xid/SXid, ECC error, PCIe GPU-loss event,
or NIC fatal/watchdog event around either incident.
- The incidents occurred on different physical workers.
- Successful retries of the exact batches argue against a fixed invalid sample
or deterministic shape-only failure. Runtime values such as selected top-k
indices may still differ because the training trajectories are not bit-exact.
- Ray subsequently failed to serialize the native TVM-FFI exception with
`Type ffi.Error does not support ToJSONGraph`; we consider this secondary.

### Related upstream work

- [#6848](https://github.com/NVIDIA/Megatron-LM/pull/6848) fixes per-microbatch
CuTe recompilation in CSA layout launches and per-sequence-length cuDNN DSA
backward recompilation. These correspond closely to the two failing call
sites, but the PR does not mention either CUDA runtime error.
- [#6206](https://github.com/NVIDIA/Megatron-LM/pull/6206) hardens general
packed-THD + CP layout/scorer handling.
- [#6910](https://github.com/NVIDIA/Megatron-LM/pull/6910) and
[#6939](https://github.com/NVIDIA/Megatron-LM/pull/6939) handle empty top-k
rows in DSA sparse-attention backward.
- [#7169](https://github.com/NVIDIA/Megatron-LM/pull/7169) reports
nondeterministic fused CSA top-k selection under ties, although it explicitly
does not cover CP > 1.
- [#5643](https://github.com/NVIDIA/Megatron-LM/issues/5643) discusses other
DeepSeek-V4 SFT + CP failures and long-context fused-DSA JIT behavior, but
reports different errors.

We have not reproduced this on
[`3703d4e33a3a2b2d11ebcc8e41f45af7ce7d1eda`](https://github.com/NVIDIA/Megatron-LM/commit/3703d4e33a3a2b2d11ebcc8e41f45af7ce7d1eda).
That revision substantially refactors the relevant DeepSeek-V4 code:
`csa_cp_layout_kernels.py` is no longer present and the cuDNN DSA integration
has moved.

Questions:

1. Are #6848, #6206, or #6910/#6939 expected to address either runtime failure?
2. Is the listed cuDNN Frontend/CUTLASS DSL/cuDNN combination supported for
fused DeepSeek-V4 training on SM90 with packed THD and CP32?
3. Which current MCore dev/main revision should be used to retest this path?
4. On recurrence, which metadata would be most useful to capture: tensor
shapes/strides, causal offsets, top-k ranges and lengths, compile-cache keys,
or CUDA stream IDs?

**Expected behavior**

Fused DeepSeek-V4 CSA/DSA training should run without intermittent CUDA runtime
errors for valid packed-THD inputs.

Start with the traceback locations in dsa_kernels.py, deepseek_v4_hybrid_attention.py, csa.py, and csa_cp_layout_kernels.py, then compare the related changes in #6848, #6206, and #6910/#6939. Retest the reported packed-THD, CP32 configuration on a current MCore revision; done means the intermittent CUDA runtime failures are reproduced or ruled out with useful metadata and the relevant upstream fixes are identified.

Written by the indexing model from the issue text.