单机多卡报索引错误

同样脚本，在单机单卡时没问题，单机多卡时，在一个epoch即将结束时报错，如下
File "/home/jovyan/tasks/training_dataset.py", line 107, in collater
return default_collate(samples)
File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/_utils/collate.py", line 128, in default_collate
elem = batch[0]
IndexError: list index out of range

调用脚本：
python -m torch.distributed.launch --nproc_per_node=3 --master_port=10096 /opt/conda/bin/unicore-train {data_path} --user-dir {user_dir} --train-subset train --valid-subset valid \
--num-workers 8 --ddp-backend=no_c10d \
--task {task} --loss {loss_func} --arch {arch} \
--optimizer adam --adam-betas '(0.9, 0.99)' --adam-eps 1e-6 --clip-norm 1.0 \
--lr-scheduler polynomial_decay --lr {lr} --warmup-ratio {warmup} --max-epoch {epoch} --batch-size {local_batch_size} \
--update-freq {update_freq} --seed {seed} \
--log-interval 100 --log-format simple \
--validate-interval 1 --keep-last-epochs 10 \
--best-checkpoint-metric {metric} --patience 20 \
--save-dir {save_dir}

备注：
1 ddp-backend=c10d提示错误，并建议改成no_c10d
2 training_dataset.py,dataset中collater使用torch的默认实现，如下
from torch.utils.data.dataloader import default_collate
def collater(self, samples):
return default_collate(samples)
__getitem__ 返回逗号分割的tensor对象，如下
return (
torch.LongTensor(ligand_data["ligand_token"]),
torch.LongTensor(ligand_encoder_data["ligand_encoder_coords"]),
torch.LongTensor(ligand_encoder_data["ligand_encoder_token"]),
torch.FloatTensor(ligand_encoder_data["ligand_mask"]),
torch.LongTensor(ligand_data["ligand_coords"]),
torch.LongTensor(ligand_data["root_idxs"]),
torch.LongTensor(ligand_data["root_root_idxs"]),
torch.LongTensor(ligand_data["root_root_root_idxs"]),
torch.LongTensor(ligand_data["theta"]),
torch.LongTensor(ligand_data["dist"]),
torch.LongTensor(ligand_data["degree"])
'

Start with training_dataset.py at line 107 and inspect how collater passes samples to torch's default_collate. Reproduce the reported command with --nproc_per_node=3 and compare the end-of-epoch behavior with single-card execution. Done means multi-card training completes the epoch without the reported IndexError.

Written by the indexing model from the issue text.