qwen3 vl reranker8b多卡训练卡住，单卡训练训一会崩掉，如何解决

Checklist / 检查清单

• I have searched existing issues, and this is a new question or discussion topic. / 我已经搜索过现有的 issues，确认这是一个新的问题与讨论。

Question Description / 问题描述

多卡训练，训练一会就会卡住，但是查看gpu利用率也是100%，但没输出，也没checkpoint保存，就一直停在这

训练代码如下：
IMAGE_MAX_TOKEN_NUM=768 \ MAX_POSITIVE_SAMPLES=10 \ MAX_NEGATIVE_SAMPLES=10 \ CUDA_VISIBLE_DEVICES=0,1,2,3 \ NPROC_PER_NODE=4 \ swift sft \ --model "/home/workspace/models/Qwen/Qwen3-VL-Reranker-8B" \ --task_type "generative_reranker" \ --loss_type "listwise_reranker" \ --tuner_type "lora" \ --lora_rank 8 \ --lora_alpha 32 \ --learning_rate "5e-5" \ --target_modules all-linear \ --dataset "my.json" \ --attn_impl flash_attn \ --padding_free true \ --torch_dtype bfloat16 \ --load_from_cache_file true \ --split_dataset_ratio 0.02 \ --eval_strategy steps \ --output_dir output \ --save_steps 50 \ --eval_steps 50 \ --logging_steps 5 \ --num_train_epochs 3 \ --max_length 4096 \ --per_device_train_batch_size 1 \ --per_device_eval_batch_size 1 \ --gradient_accumulation_steps 8 \ --dataloader_num_workers 4 \ --dataset_num_proc 4 \ --warmup_ratio 0.05 \ --dataloader_drop_last true

单卡训练，训练到中间就会显示

{'loss': 9.47974472, 'grad_norm': 43.4076004, 'learning_rate': 4.694e-05, 'acc': 0.54285714, 'epoch': 0.6, 'global_step/max_steps': '3695/18357', 'percentage': '20.13%', 'elapsed_time': '22h 50m 23s', 'remaining_time': '3d 18h 37m 49s', 'memory(GiB)': 76.77, 'train_speed(iter/s)': 0.044938}

Train:  20%|██        | 3695/18357 [22:50:23<87:44:53, 21.55s/it]
Train:  20%|██        | 3695/18357 [22:50:23<87:44:53, 21.55s/it]
Train:  20%|██        | 3696/18357 [22:50:43<86:00:42, 21.12s/it]
Train:  20%|██        | 3697/18357 [22:51:07<89:02:16, 21.86s/it]
Train:  20%|██        | 3698/18357 [22:51:24<82:36:03, 20.29s/it]
Train:  20%|██        | 3699/18357 [22:51:52<91:52:36, 22.56s/it]
Train:  20%|██        | 3700/18357 [22:52:12<88:42:02, 21.79s/it]
                                                                 
{'loss': 6.85271835, 'grad_norm': 16.47181129, 'learning_rate': 4.693e-05, 'acc': 0.6984127, 'epoch': 0.6, 'global_step/max_steps': '3700/18357', 'percentage': '20.16%', 'elapsed_time': '22h 52m 12s', 'remaining_time': '3d 18h 35m 46s', 'memory(GiB)': 76.77, 'train_speed(iter/s)': 0.04494}

Train:  20%|██        | 3700/18357 [22:52:12<88:42:02, 21.79s/it]
Train:  20%|██        | 3700/18357 [22:52:12<88:42:02, 21.79s/it][INFO:swift] last_model_checkpoint: /home/workspace/experiments/ms-swift/reranker/qwen3vl/8b/lora/qwen3_vl_8b_l8/v1-20260202-023111/checkpoint-3700
[INFO:swift] best_model_checkpoint: /home/workspace/experiments/ms-swift/reranker/qwen3vl/8b/lora/qwen3_vl_8b_l8/v1-20260202-023111/checkpoint-3000
[INFO:swift] images_dir: /home/workspace/experiments/ms-swift/reranker/qwen3vl/8b/lora/qwen3_vl_8b_l8/v1-20260202-023111/images
[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/workspace/github/official/ms-swift/swift/cli/sft.py", line 20, in <module>
[rank0]:     sft_main()
[rank0]:   File "/home/workspace/github/official/ms-swift/swift/pipelines/train/sft.py", line 346, in sft_main
[rank0]:     return SwiftSft(args).main()
[rank0]:   File "/home/workspace/github/official/ms-swift/swift/pipelines/base.py", line 47, in main
[rank0]:     result = self.run()
[rank0]:   File "/home/workspace/github/official/ms-swift/swift/ray/base.py", line 169, in wrapper
[rank0]:     return func(self, *args, **kwargs)
[rank0]:   File "/home/workspace/github/official/ms-swift/swift/pipelines/train/sft.py", line 198, in run
[rank0]:     return self.train(trainer)
[rank0]:   File "/home/workspace/github/official/ms-swift/swift/pipelines/train/sft.py", line 264, in train
[rank0]:     trainer.train(resume_checkpoint)
[rank0]:   File "/home/workspace/github/official/ms-swift/swift/trainers/trainer.py", line 65, in train
[rank0]:     return super().train(*args, **kwargs)
[rank0]:   File "/home/workspace/github/official/ms-swift/swift/trainers/mixin.py", line 936, in train
[rank0]:     res = super().train(*args, **kwargs)
[rank0]:   File "/home/envs/miniconda3/envs/llm_cls/lib/python3.10/site-packages/transformers/trainer.py", line 2325, in train
[rank0]:     return inner_training_loop(
[rank0]:   File "/home/envs/miniconda3/envs/llm_cls/lib/python3.10/site-packages/transformers/trainer.py", line 2756, in _inner_training_loop
[rank0]:     self._maybe_log_save_evaluate(
[rank0]:   File "/home/workspace/github/official/ms-swift/swift/trainers/mixin.py", line 1009, in _maybe_log_save_evaluate
[rank0]:     super()._maybe_log_save_evaluate(tr_loss, *args, **kwargs)
[rank0]:   File "/home/envs/miniconda3/envs/llm_cls/lib/python3.10/site-packages/transformers/trainer.py", line 3228, in _maybe_log_save_evaluate
[rank0]:     self._save_checkpoint(model, trial)
[rank0]:   File "/home/workspace/github/official/ms-swift/swift/trainers/mixin.py", line 564, in _save_checkpoint
[rank0]:     result = super()._save_checkpoint(*args, **kwargs)
[rank0]:   File "/home/envs/miniconda3/envs/llm_cls/lib/python3.10/site-packages/transformers/trainer.py", line 3325, in _save_checkpoint
[rank0]:     self.save_model(output_dir, _internal_call=True)
[rank0]:   File "/home/envs/miniconda3/envs/llm_cls/lib/python3.10/site-packages/transformers/trainer.py", line 4212, in save_model
[rank0]:     state_dict = self.accelerator.get_state_dict(self.deepspeed)
[rank0]:   File "/home/envs/miniconda3/envs/llm_cls/lib/python3.10/site-packages/accelerate/accelerator.py", line 3495, in get_state_dict
[rank0]:     state_dict = clone_tensors_for_torch_save(self.unwrap_model(model).state_dict())
[rank0]:   File "/home/envs/miniconda3/envs/llm_cls/lib/python3.10/site-packages/deepspeed/checkpoint/utils.py", line 65, in clone_tensors_for_torch_save
[rank0]:     return type(item)({k: clone_tensors_for_torch_save(v, device) for k, v in item.items()})
[rank0]:   File "/home/envs/miniconda3/envs/llm_cls/lib/python3.10/site-packages/deepspeed/checkpoint/utils.py", line 65, in <dictcomp>
[rank0]:     return type(item)({k: clone_tensors_for_torch_save(v, device) for k, v in item.items()})
[rank0]:   File "/home/envs/miniconda3/envs/llm_cls/lib/python3.10/site-packages/deepspeed/checkpoint/utils.py", line 59, in clone_tensors_for_torch_save
[rank0]:     return item.detach().to(device)
[rank0]:   File "/home/envs/miniconda3/envs/llm_cls/lib/python3.10/site-packages/torch/utils/data/_utils/signal_handling.py", line 73, in handler
[rank0]:     _error_if_any_worker_fails()
[rank0]: RuntimeError: DataLoader worker (pid 511) is killed by signal: Killed. 
Train:  20%|██        | 3700/18357 [22:52:22<90:36:27, 22.25s/it]
[rank0]:[W203 01:23:57.289475893 ProcessGroupNCCL.cpp:1538] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
E0203 01:24:01.380000 184 site-packages/torch/distributed/elastic/multiprocessing/api.py:874] failed (exitcode: 1) local_rank: 0 (pid: 249) of binary: /home/envs/miniconda3/envs/llm_cls/bin/python
Traceback (most recent call last):
  File "/home/envs/miniconda3/envs/llm_cls/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/envs/miniconda3/envs/llm_cls/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/envs/miniconda3/envs/llm_cls/lib/python3.10/site-packages/torch/distributed/run.py", line 905, in <module>
    main()
  File "/home/envs/miniconda3/envs/llm_cls/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 357, in wrapper
    return f(*args, **kwargs)
  File "/home/envs/miniconda3/envs/llm_cls/lib/python3.10/site-packages/torch/distributed/run.py", line 901, in main
    run(args)
  File "/home/envs/miniconda3/envs/llm_cls/lib/python3.10/site-packages/torch/distributed/run.py", line 892, in run
    elastic_launch(
  File "/home/envs/miniconda3/envs/llm_cls/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 143, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/envs/miniconda3/envs/llm_cls/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 277, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/home/workspace/github/official/ms-swift/swift/cli/sft.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2026-02-03_01:24:01
  host      : vlm-pred-a100-1-2-2n4vg
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 249)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================

========== 训练结束 ==========
退出状态: 0
Tue Feb  3 01:24:02 UTC 2026```

---
上述数据集的构造格式均如下：
`[
    {
        "messages": [
            {
                "role": "user",
                "content": "prompt"
            }
        ],
        "images": [],
        "positive_messages": [
            [
                {
                    "role": "assistant",
                    "content": "<image>"
                }
            ],
            [
                {
                    "role": "assistant",
                    "content": "<image>"
                }
            ]
        ],
        "positive_images": [
            [
                "img_url"
            ],
            [
                "img_url"
            ]
        ],
        "negative_messages": [
            [
                {
                    "role": "assistant",
                    "content": "<image>"
                }
            ],
            [
                {
                    "role": "assistant",
                    "content": "<image>"
                }
            ],
            [
                {
                    "role": "assistant",
                    "content": "<image>"
                }
            ]
        ],
        "negative_images": [
            [
                "img_url"
            ],
            [
                "img_url"
            ],
            [
                "img_url"
            ]
        ]
    },
    ...
]`

Start with the provided swift sft command and traceback, especially swift/cli/sft.py, swift/pipelines/train/sft.py, swift/trainers/mixin.py, and the Transformers save path. Reproduce with the supplied dataset shape and training settings, then inspect the DataLoader worker failure during checkpoint saving. Done means the training run no longer hangs or exits with a killed worker and checkpoints are saved successfully.