An error occurred during training using Protenix Docker 1.0.0.4

Dear Protenix Development Team，my training was conducted using the official dataset, with the mirroring used being ai4s-share-public-cn-beijing.cr.volces.com/release/protenix:1.0.0.4, and this error can be reproduced every time.

My training environment is Driver Version: 560.35.03 CUDA Version: 12.6, and the graphics card is H 100

The training instruction is：
```
torchrun --nproc_per_node 8 \
--master_port=29506 \
./runner/train.py \
--run_name protenix_train \
--model_name "protenix_base_default_v1.0.0" \
--seed 42 \
--base_dir ./output/ \
--dtype bf16 \
--project protenix \
--use_wandb false \
--diffusion_batch_size 16 \
--iters_to_accumulate 2 \
--eval_interval 400 \
--log_interval 50 \
--checkpoint_interval 2000 \
--ema_decay 0.999 \
--train_crop_size 768 \
--max_steps 100000 \
--warmup_steps 1000 \
--lr 0.0005 \
--model.N_cycle 4 \
--sample_diffusion.N_step 20 \
--triangle_attention "cuequivariance" \
--triangle_multiplicative "cuequivariance" \
--data.train_sets weightedPDB_before2109_wopb_nometalc_0925 \
--data.test_sets recentPDB_1536_sample384_0925,posebusters_0925 \
--data.posebusters_0925.base_info.max_n_token 768 \
--loss.weight.smooth_lddt 0.0 \
--loss.weight.alpha_bond 1.0
```

The error is as shown in the following message. How should I troubleshoot the cause of this error? Thank you!
@zhangyuxuann
```
2026-06-03 02:45:41,755 [/app/code/./runner/train.py:468] INFO root: Step 3599, eval posebusters_0925: {'posebusters_0925/ema0.999_resolved_loss.avg': np.float64(0.23424069203791165), 'posebusters_0925/ema0.9
99_weighted_pde_loss.avg': np.float64(0.00033150770981419124), 'posebusters_0925/ema0.999_pde_loss.avg': np.float64(3.315077184495472), 'posebusters_0925/ema0.999_loss.avg': np.float64(517.6821294239589), 'po
sebusters_0925/ema0.999_weighted_smooth_lddt_loss.avg': np.float64(0.0), 'posebusters_0925/ema0.999_pae_loss.avg': np.float64(4.15888345128014), 'posebusters_0925/ema0.999_weighted_resolved_loss.avg': np.floa
t64(2.342406854343911e-05), 'posebusters_0925/ema0.999_lddt/complex/gpde.rank1.avg': np.float64(0.3719116247835613), 'posebusters_0925/ema0.999_lddt/complex/mean.avg': np.float64(0.36325206678538097), 'posebu
sters_0925/ema0.999_smooth_lddt_loss.avg': np.float64(0.6211075533004035), 'posebusters_0925/ema0.999_lddt/complex/plddt.rank1.avg': np.float64(0.3640599228086926), 'posebusters_0925/ema0.999_lddt/complex/wor
st.avg': np.float64(0.34535366679940904), 'posebusters_0925/ema0.999_weighted_bond_loss.avg': np.float64(2.3117294575486866), 'posebusters_0925/ema0.999_plddt_loss.avg': np.float64(3.565737787882487), 'posebu
sters_0925/ema0.999_weighted_mse_loss.avg': np.float64(515.3696882702055), 'posebusters_0925/ema0.999_lddt/complex/median.avg': np.float64(0.3634037167543457), 'posebusters_0925/ema0.999_weighted_pae_loss.avg
': np.float64(0.0), 'posebusters_0925/ema0.999_lddt/complex/best.avg': np.float64(0.38063443906250455), 'posebusters_0925/ema0.999_lddt/complex/random.avg': np.float64(0.36405437333243235), 'posebusters_0925/
ema0.999_lddt/complex/ranking_score.rank1.avg': np.float64(0.3638552948832512), 'posebusters_0925/ema0.999_mse_loss.avg': np.float64(128.84242206755138), 'posebusters_0925/ema0.999_weighted_plddt_loss.avg': n
p.float64(0.0003565737704602292), 'posebusters_0925/ema0.999_bond_loss.avg': np.float64(0.5779323643871717)}
[step 3599: 799/800] : 100%|________________________________________________________________________________________________________________________________________________| 799/800 [3:55:17<00:17, 17.67s/it]
0%| | 0/800 [00:00, std::allocator >) + 0x98 (0x7fb53fb785e8 in /root/miniconda3/lib/python3.11/site-packages/torch/
lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional > >) + 0x23d (0x7fb4ea5cba6d in /root/miniconda3/lib/python3.11/site-packages/torch/lib
/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0xc80 (0x7fb4ea5cd7f0 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7fb4ea5ceefd in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #4: + 0xdbbf4 (0x7fb4da3efbf4 in /root/miniconda3/bin/../lib/libstdc++.so.6)
frame #5: + 0x94ac3 (0x7fb540e48ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #6: clone + 0x44 (0x7fb540ed9a74 in /usr/lib/x86_64-linux-gnu/libc.so.6)

W0603 04:19:17.128000 104 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 106 closing signal SIGTERM
W0603 04:19:17.133000 104 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 108 closing signal SIGTERM
W0603 04:19:17.134000 104 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 109 closing signal SIGTERM
[rank4]:[E603 04:19:17.315831399 ProcessGroupNCCL.cpp:1685] [PG ID 0 PG GUID 0(default_pg) Rank 4] Observed flight recorder dump signal from another rank via TCPStore.
W0603 04:19:17.135000 104 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 110 closing signal SIGTERM
W0603 04:19:17.137000 104 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 111 closing signal SIGTERM
W0603 04:19:17.138000 104 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 112 closing signal SIGTERM
W0603 04:19:17.139000 104 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 113 closing signal SIGTERM
E0603 04:19:17.163000 104 site-packages/torch/distributed/elastic/multiprocessing/api.py:874] failed (exitcode: -6) local_rank: 1 (pid: 107) of binary: /root/miniconda3/bin/python3.11
Traceback (most recent call last):
File "/root/miniconda3/bin/torchrun", line 8, in
sys.exit(main())
^^^^^^
File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/run.py", line 892, in main
run(args)
File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/run.py", line 883, in run
elastic_launch(
File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 139, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 270, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
====================================================
./runner/train.py FAILED
----------------------------------------------------
Failures:

----------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2026-06-03_04:19:17
host : d69c7cf49a03
rank : 1 (local_rank: 1)
exitcode : -6 (pid: 107)
error_file:
traceback : Signal 6 (SIGABRT) received by PID 107
====================================================

```

Start with the distributed training entry point `runner/train.py` and the reported NCCL watchdog timeout during an `ALLREDUCE` at step 3749; the logs also show training and evaluation continuing beforehand. Check how distributed ranks reach this collective and compare their progress around the timeout. Done means identifying a reproducible cause or a focused diagnostic that explains the rank stall; the issue provides no test or fix criteria.

Written by the indexing model from the issue text.