sft微调qwen3.5-27b，参考微调脚本，基于latex ocr数据 训练进度一半的报torch.distributed.elastic.multiprocessing.api.SignalException: Process 3952512 got signal: 1

Checklist / 检查清单

• I have searched existing issues, and this is a new bug report. / 我已经搜索过现有的 issues，确认这是一个新的 bug report。

Bug Description / Bug 描述

Train: 52%|█████▏ | 2028/3872 [4:11:42<3:12:59, 6.28s/it]
Train: 52%|█████▏ | 2029/3872 [4:11:48<3:14:53, 6.34s/it]
Train: 52%|█████▏ | 2030/3872 [4:11:55<3:16:06, 6.39s/it]

{'loss': '0.2753', 'grad_norm': '0.3682', 'learning_rate': '5.013e-05', 'token_acc': '0.9245', 'epoch': '0.5243', 'global_step/max_steps': '2030/3872', 'elapsed_time': '4h 11m 55s', 'remaining_time': '3h 48m 36s', 'memory(GiB)': '64.42', 'train_speed(s/it)': '7.446'}

Train: 52%|█████▏ | 2030/3872 [4:11:55<3:16:06, 6.39s/it]
Train: 52%|█████▏ | 2030/3872 [4:11:55<3:16:06, 6.39s/it]
Train: 52%|█████▏ | 2031/3872 [4:12:01<3:13:50, 6.32s/it]
Train: 52%|█████▏ | 2032/3872 [4:12:08<3:15:26, 6.37s/it]
Train: 53%|█████▎ | 2033/3872 [4:12:14<3:15:50, 6.39s/it]
Train: 53%|█████▎ | 2034/3872 [4:12:20<3:15:33, 6.38s/it]
Train: 53%|█████▎ | 2035/3872 [4:12:27<3:13:29, 6.32s/it]

{'loss': '0.3809', 'grad_norm': '0.3813', 'learning_rate': '4.991e-05', 'token_acc': '0.9011', 'epoch': '0.5256', 'global_step/max_steps': '2035/3872', 'elapsed_time': '4h 12m 27s', 'remaining_time': '3h 47m 53s', 'memory(GiB)': '64.42', 'train_speed(s/it)': '7.443'}

Train: 53%|█████▎ | 2035/3872 [4:12:27<3:13:29, 6.32s/it]
Train: 53%|█████▎ | 2035/3872 [4:12:27<3:13:29, 6.32s/it]
Train: 53%|█████▎ | 2036/3872 [4:12:33<3:15:16, 6.38s/it]
Train: 53%|█████▎ | 2037/3872 [4:12:40<3:15:57, 6.41s/it]
Train: 53%|█████▎ | 2038/3872 [4:12:46<3:15:59, 6.41s/it]
Train: 53%|█████▎ | 2039/3872 [4:12:52<3:16:33, 6.43s/it]
Train: 53%|█████▎ | 2040/3872 [4:12:59<3:13:49, 6.35s/it]

{'loss': '0.2386', 'grad_norm': '0.4714', 'learning_rate': '4.97e-05', 'token_acc': '0.9353', 'epoch': '0.5269', 'global_step/max_steps': '2040/3872', 'elapsed_time': '4h 12m 59s', 'remaining_time': '3h 47m 11s', 'memory(GiB)': '64.42', 'train_speed(s/it)': '7.441'}

Train: 53%|█████▎ | 2040/3872 [4:12:59<3:13:49, 6.35s/it]
Train: 53%|█████▎ | 2040/3872 [4:12:59<3:13:49, 6.35s/it]
Train: 53%|█████▎ | 2041/3872 [4:13:05<3:14:48, 6.38s/it]
Train: 53%|█████▎ | 2042/3872 [4:13:11<3:12:32, 6.31s/it]
Train: 53%|█████▎ | 2043/3872 [4:13:18<3:15:14, 6.40s/it]
Train: 53%|█████▎ | 2044/3872 [4:13:24<3:15:25, 6.41s/it]
Train: 53%|█████▎ | 2045/3872 [4:13:31<3:14:54, 6.40s/it]

{'loss': '0.2885', 'grad_norm': '0.4223', 'learning_rate': '4.949e-05', 'token_acc': '0.9225', 'epoch': '0.5282', 'global_step/max_steps': '2045/3872', 'elapsed_time': '4h 13m 31s', 'remaining_time': '3h 46m 30s', 'memory(GiB)': '64.42', 'train_speed(s/it)': '7.438'}

Train: 53%|█████▎ | 2045/3872 [4:13:31<3:14:54, 6.40s/it]
Train: 53%|█████▎ | 2045/3872 [4:13:31<3:14:54, 6.40s/it]
Train: 53%|█████▎ | 2046/3872 [4:13:37<3:15:14, 6.42s/it]
Train: 53%|█████▎ | 2047/3872 [4:13:44<3:15:28, 6.43s/it]W0312 21:33:06.886000 3952512 /appdata/zhangkailin/envs/swift3/lib/python3.12/site-packages/torch/distributed/elastic/agent/server/api.py:739] Received 1 death signal, shutting down workers
W0312 21:33:06.888000 3952512 /appdata/zhangkailin/envs/swift3/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/api.py:1010] Sending process 3952579 closing signal SIGHUP
W0312 21:33:06.889000 3952512 /appdata/zhangkailin/envs/swift3/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/api.py:1010] Sending process 3952580 closing signal SIGHUP
W0312 21:33:06.890000 3952512 /appdata/zhangkailin/envs/swift3/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/api.py:1010] Sending process 3952581 closing signal SIGHUP
W0312 21:33:06.890000 3952512 /appdata/zhangkailin/envs/swift3/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/api.py:1010] Sending process 3952582 closing signal SIGHUP
W0312 21:33:06.891000 3952512 /appdata/zhangkailin/envs/swift3/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/api.py:1010] Sending process 3952583 closing signal SIGHUP
W0312 21:33:06.892000 3952512 /appdata/zhangkailin/envs/swift3/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/api.py:1010] Sending process 3952584 closing signal SIGHUP
W0312 21:33:06.893000 3952512 /appdata/zhangkailin/envs/swift3/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/api.py:1010] Sending process 3952585 closing signal SIGHUP
W0312 21:33:06.894000 3952512 /appdata/zhangkailin/envs/swift3/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/api.py:1010] Sending process 3952586 closing signal SIGHUP
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/home/zhangkailin/.conda/envs/swift3/lib/python3.12/site-packages/torch/distributed/run.py", line 995, in
main()
File "/home/zhangkailin/.conda/envs/swift3/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 362, in wrapper
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/home/zhangkailin/.conda/envs/swift3/lib/python3.12/site-packages/torch/distributed/run.py", line 991, in main
run(args)
File "/home/zhangkailin/.conda/envs/swift3/lib/python3.12/site-packages/torch/distributed/run.py", line 982, in run
elastic_launch(
File "/home/zhangkailin/.conda/envs/swift3/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 170, in call
return launch_agent(self._config, self._entrypoint, list(args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/zhangkailin/.conda/envs/swift3/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 308, in launch_agent
result = agent.run()
^^^^^^^^^^^
File "/home/zhangkailin/.conda/envs/swift3/lib/python3.12/site-packages/torch/distributed/elastic/metrics/api.py", line 134, in wrapper
result = f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/home/zhangkailin/.conda/envs/swift3/lib/python3.12/site-packages/torch/distributed/elastic/agent/server/api.py", line 731, in run
result = self._invoke_run(role)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/zhangkailin/.conda/envs/swift3/lib/python3.12/site-packages/torch/distributed/elastic/agent/server/api.py", line 908, in _invoke_run
time.sleep(monitor_interval)
File "/home/zhangkailin/.conda/envs/swift3/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 86, in _terminate_process_handler
raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
torch.distributed.elastic.multiprocessing.api.SignalException: Process 3952512 got signal: 1

How to Reproduce / 如何复现

环境 8卡A100 80G

PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True'
NPROC_PER_NODE=8
MAX_PIXELS=1003520
VIDEO_MAX_PIXELS=50176
FPS_MAX_FRAMES=12
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
swift sft
--model /appdata/zhangkailin/models/Qwen/Qwen3.5-27B
--tuner_type lora
--dataset '/appdata/zhangkailin/datasets/AI-ModelScope___la_te_x_ocr'
'/appdata/zhangkailin/datasets/AI-ModelScope___alpaca-gpt4-data-zh'
--load_from_cache_file true
--add_non_thinking_prefix true
--loss_scale ignore_empty_think
--split_dataset_ratio 0.01
--torch_dtype bfloat16
--num_train_epochs 1
--per_device_train_batch_size 4
--per_device_eval_batch_size 4
--learning_rate 1e-4
--lora_rank 8
--lora_alpha 32
--target_modules all-linear
--gradient_accumulation_steps 1
--group_by_length true
--output_dir output/Qwen3.5-32B
--eval_steps 50
--save_steps 50
--save_total_limit 2
--logging_steps 5
--max_length 2048
--warmup_ratio 0.05
--dataset_num_proc 4
--dataloader_num_workers 4
--deepspeed zero2
--model_author swift
--model_name swift-robot
--report_to wandb

Additional Information / 补充信息

No response

Start with the provided 8-GPU swift sft command and inspect the complete logs around step 2047, including the source of the received death signal. No project file or test is named; reproduce the run if possible and determine whether the interruption comes from the training process or its environment. Done means the cause is identified and training can proceed reliably past the failure point, with a regression test or documented reproduction if applicable.