About issues running megatritonkernel

Hello, I encountered an NCCL error during testing. I had successfully run megatritonkernel before, but suddenly encountered this problem. Could you please provide some possible solutions? Thank you very much.
The following is the error message：
Found NVSHMEM_HOME from Python nvidia-nvshmem-cu12: /usr/local/lib/python3.12/dist-packages/nvidia/nvshmem
torchrun --node_rank=0 --nproc_per_node=8 --nnodes=1 --rdzv_endpoint=127.0.0.1:23456 --nproc_per_node 4 python/triton_dist/mega_triton_kernel/test/models/bench_qwen3.py --model /workspace/weight/Qwen3-8B --seq_len 512
W0819 10:35:47.038000 1691143 torch/distributed/run.py:766]
W0819 10:35:47.038000 1691143 torch/distributed/run.py:766] *****************************************
W0819 10:35:47.038000 1691143 torch/distributed/run.py:766] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
W0819 10:35:47.038000 1691143 torch/distributed/run.py:766] *****************************************
[2025-08-19 10:35:52,216] [INFO] [real_accelerator.py:254:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-08-19 10:35:52,220] [INFO] [real_accelerator.py:254:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-08-19 10:35:52,296] [INFO] [real_accelerator.py:254:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-08-19 10:35:52,302] [INFO] [real_accelerator.py:254:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-08-19 10:35:53,727] [INFO] [logging.py:107:log_dist] [Rank -1] [TorchCheckpointEngine] Initialized with serialization = False
[2025-08-19 10:35:53,727] [INFO] [logging.py:107:log_dist] [Rank -1] [TorchCheckpointEngine] Initialized with serialization = False
[2025-08-19 10:35:53,851] [INFO] [logging.py:107:log_dist] [Rank -1] [TorchCheckpointEngine] Initialized with serialization = False
[2025-08-19 10:35:53,857] [INFO] [logging.py:107:log_dist] [Rank -1] [TorchCheckpointEngine] Initialized with serialization = False
Using flash_attn, which is much slower than flash_attn_interface for sm90Using flash_attn, which is much slower than flash_attn_interface for sm90

Using flash_attn, which is much slower than flash_attn_interface for sm90
Using flash_attn, which is much slower than flash_attn_interface for sm90
[rank3]:[W819 10:35:54.455718714 ProcessGroupNCCL.cpp:4751] [PG ID 1 PG GUID 1 Rank 3] using GPU 3 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can pecify device_id in init_process_group() to force use of a particular device.
[rank2]:[W819 10:35:54.455811075 ProcessGroupNCCL.cpp:4751] [PG ID 1 PG GUID 1 Rank 2] using GPU 2 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can pecify device_id in init_process_group() to force use of a particular device.
[rank1]:[W819 10:35:54.456257199 ProcessGroupNCCL.cpp:4751] [PG ID 1 PG GUID 1 Rank 1] using GPU 1 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can pecify device_id in init_process_group() to force use of a particular device.
[rank0]:[W819 10:35:54.711103324 ProcessGroupNCCL.cpp:4751] [PG ID 1 PG GUID 1 Rank 0] using GPU 0 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can pecify device_id in init_process_group() to force use of a particular device.
[rank3]: Traceback (most recent call last):
[rank3]: File "/workspace/Triton-distributed/python/triton_dist/mega_triton_kernel/test/models/bench_qwen3.py", line 83, in
[rank3]: TP_GROUP = initialize_distributed(seed=0)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/workspace/Triton-distributed/python/triton_dist/utils.py", line 199, in initialize_distributed
[rank3]: torch.distributed.barrier(_TP_GROUP)
[rank3]: File "/usr/local/lib/python3.12/dist-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
[rank3]: return func(*args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/usr/local/lib/python3.12/dist-packages/torch/distributed/distributed_c10d.py", line 4657, in barrier
[rank3]: work = group.barrier(opts=opts)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: torch.distributed.DistBackendError: NCCL error in: /opt/pytorch/pytorch/torch/csrc/distributed/c10d/NCCLUtils.cpp:77, unhandled cuda error (run with NCCL_DEBUG=INFO for details), NCCL version 2.26.3
[rank3]: ncclUnhandledCudaError: Call to CUDA function failed.
[rank3]: Last error:
[rank3]: Cuda failure 802 'system not yet initialized'
[rank2]: Traceback (most recent call last):
[rank2]: File "/workspace/Triton-distributed/python/triton_dist/mega_triton_kernel/test/models/bench_qwen3.py", line 83, in
[rank2]: TP_GROUP = initialize_distributed(seed=0)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/workspace/Triton-distributed/python/triton_dist/utils.py", line 199, in initialize_distributed
[rank2]: torch.distributed.barrier(_TP_GROUP)
[rank2]: File "/usr/local/lib/python3.12/dist-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
[rank2]: return func(*args, **kwargs)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/usr/local/lib/python3.12/dist-packages/torch/distributed/distributed_c10d.py", line 4657, in barrier
[rank2]: work = group.barrier(opts=opts)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: torch.distributed.DistBackendError: NCCL error in: /opt/pytorch/pytorch/torch/csrc/distributed/c10d/NCCLUtils.cpp:77, unhandled cuda error (run with NCCL_DEBUG=INFO for details), NCCL version 2.26.3
[rank2]: ncclUnhandledCudaError: Call to CUDA function failed.
[rank2]: Last error:
[rank2]: Cuda failure 802 'system not yet initialized'
[rank1]: Traceback (most recent call last):
[rank1]: File "/workspace/Triton-distributed/python/triton_dist/mega_triton_kernel/test/models/bench_qwen3.py", line 83, in
[rank1]: TP_GROUP = initialize_distributed(seed=0)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/workspace/Triton-distributed/python/triton_dist/utils.py", line 199, in initialize_distributed
[rank1]: torch.distributed.barrier(_TP_GROUP)
[rank1]: File "/usr/local/lib/python3.12/dist-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
[rank1]: return func(*args, **kwargs)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/usr/local/lib/python3.12/dist-packages/torch/distributed/distributed_c10d.py", line 4657, in barrier
[rank1]: work = group.barrier(opts=opts)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: torch.distributed.DistBackendError: NCCL error in: /opt/pytorch/pytorch/torch/csrc/distributed/c10d/NCCLUtils.cpp:77, unhandled cuda error (run with NCCL_DEBUG=INFO for details), NCCL version 2.26.3
[rank1]: ncclUnhandledCudaError: Call to CUDA function failed.
[rank1]: Last error:
[rank1]: Cuda failure 802 'system not yet initialized'
[rank0]: Traceback (most recent call last):
[rank0]: File "/workspace/Triton-distributed/python/triton_dist/mega_triton_kernel/test/models/bench_qwen3.py", line 83, in
[rank0]: TP_GROUP = initialize_distributed(seed=0)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/workspace/Triton-distributed/python/triton_dist/utils.py", line 199, in initialize_distributed
[rank0]: torch.distributed.barrier(_TP_GROUP)
[rank0]: File "/usr/local/lib/python3.12/dist-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
[rank0]: return func(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.12/dist-packages/torch/distributed/distributed_c10d.py", line 4657, in barrier
[rank0]: work = group.barrier(opts=opts)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: torch.distributed.DistBackendError: NCCL error in: /opt/pytorch/pytorch/torch/csrc/distributed/c10d/NCCLUtils.cpp:77, unhandled cuda error (run with NCCL_DEBUG=INFO for details), NCCL version 2.26.3
[rank0]: ncclUnhandledCudaError: Call to CUDA function failed.
[rank0]: Last error:
[rank0]: Cuda failure 802 'system not yet initialized'
[rank0]:[W819 10:35:59.162554309 ProcessGroupNCCL.cpp:1477] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
W0819 10:36:00.161000 1691143 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 1691340 closing signal SIGTERM
W0819 10:36:00.162000 1691143 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 1691341 closing signal SIGTERM
W0819 10:36:00.163000 1691143 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 1691343 closing signal SIGTERM
E0819 10:36:00.341000 1691143 torch/distributed/elastic/multiprocessing/api.py:874] failed (exitcode: 1) local_rank: 2 (pid: 1691342) of binary: /usr/bin/python
Traceback (most recent call last):
File "/usr/local/bin/torchrun", line 33, in
sys.exit(load_entry_point('torch==2.7.0a0+79aa17489c.nv25.4', 'console_scripts', 'torchrun')())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 892, in main
run(args)
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 883, in run
elastic_launch(
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 139, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 270, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
python/triton_dist/mega_triton_kernel/test/models/bench_qwen3.py FAILED