terminate called without an active exception on multi-GPU training

Environment:

4x NVIDIA GPUs (H100/A100)
PyTorch 2.x, CUDA 12.8
NCCL 2.26.2
NCCL_IB_DISABLE=1, NCCL_P2P_DISABLE=1
Problem: Multi-GPU ShapeVAE training crashes with terminate called without an active exception during SPARSE backend initialization (flex_gemm + flash_attn). Single GPU training works fine.

Error output:

[SPARSE] Conv backend: flex_gemm; Attention backend: flash_attn
[SPARSE] Conv backend: flex_gemm; Attention backend: flash_attn
terminate called without an active exception
terminate called without an active exception
...

Observations:

DDP initialization completes successfully
Crash occurs when multiple GPUs simultaneously initialize SPARSE conv/attention backends
Appears to be a race condition in flex_gemm or flash_attn backend initialization
No Python traceback - crash happens at C++ level
Expected behavior: Multi-GPU training should work as single-GPU does.

Workaround tried (failed):

1. Setting NCCL environment variables

export NCCL_DEBUG=INFO
export NCCL_ASYNC_ERROR_HANDLING=1
export PL_TORCH_DISTRIBUTED_DEBUG=DETAIL
export NCCL_BLOCKING_WAIT=1
export NCCL_IB_DISABLE=1 
export NCCL_P2P_DISABLE=1 
export MASTER_ADDR=localhost
export MASTER_PORT=29501

2. Skipping initial snapshot

@JeffreyXiang Can u give some advice? Thanks!

Start by reproducing ShapeVAE multi-GPU training with the listed PyTorch, CUDA, NCCL, flex_gemm, and flash_attn setup, then compare it with single-GPU training. Trace SPARSE backend initialization around the reported C++-level termination; done means multi-GPU training completes without the crash.