sft 垂域 文生文 训练中断

Checklist / 检查清单

• I have searched existing issues, and this is a new bug report. / 我已经搜索过现有的 issues，确认这是一个新的 bug report。

Bug Description / Bug 描述

{'eval_loss': '3.055', 'eval_runtime': '41.2', 'eval_samples_per_second': '25.97', 'eval_steps_per_second': '3.253', 'eval_token_acc': '0.4127', 'epoch': '2.143', 'global_step/max_steps': '480/672', 'elapsed_time': '6h 31m 8s', 'remaining_time': '2h 36m 27s', 'memory(GiB)': '66.85', 'train_speed(s/it)': '48.89'}

Train: 71%|███████▏ | 480/672 [6:31:08<2:23:20, 44.79s/it]
Val: 100%|██████████| 134/134 [00:40<00:00, 3.32it/s]
Train: 71%|███████▏ | 480/672 [6:31:08<2:23:20, 44.79s/it]
Val: 100%|██████████| 134/134 [00:40<00:00, 3.29it/s]

Train: 72%|███████▏ | 481/672 [6:31:59<3:08:22, 59.17s/it]
Train: 72%|███████▏ | 482/672 [6:32:43<2:52:41, 54.53s/it]W0722 21:16:14.716000 3681165 /appdata/zhangkailin/conda/envs/swift-sft/lib/python3.12/site-packages/torch/distributed/elastic/agent/server/api.py:739] Received 1 death signal, shutting down workers
W0722 21:16:14.717000 3681165 /appdata/zhangkailin/conda/envs/swift-sft/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/api.py:1012] Sending process 3681211 closing signal SIGHUP
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/home/zhangkailin/.conda/envs/swift-sft/lib/python3.12/site-packages/torch/distributed/run.py", line 994, in
main()
File "/home/zhangkailin/.conda/envs/swift-sft/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 362, in wrapper
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/home/zhangkailin/.conda/envs/swift-sft/lib/python3.12/site-packages/torch/distributed/run.py", line 990, in main
run(args)
File "/home/zhangkailin/.conda/envs/swift-sft/lib/python3.12/site-packages/torch/distributed/run.py", line 981, in run
elastic_launch(
File "/home/zhangkailin/.conda/envs/swift-sft/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 170, in call
return launch_agent(self._config, self._entrypoint, list(args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/zhangkailin/.conda/envs/swift-sft/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 308, in launch_agent
result = agent.run()
^^^^^^^^^^^
File "/home/zhangkailin/.conda/envs/swift-sft/lib/python3.12/site-packages/torch/distributed/elastic/metrics/api.py", line 134, in wrapper
result = f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/home/zhangkailin/.conda/envs/swift-sft/lib/python3.12/site-packages/torch/distributed/elastic/agent/server/api.py", line 731, in run
result = self._invoke_run(role)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/zhangkailin/.conda/envs/swift-sft/lib/python3.12/site-packages/torch/distributed/elastic/agent/server/api.py", line 908, in _invoke_run
time.sleep(monitor_interval)
File "/home/zhangkailin/.conda/envs/swift-sft/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 86, in _terminate_process_handler
raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
torch.distributed.elastic.multiprocessing.api.SignalException: Process 3681165 got signal: 1

How to Reproduce / 如何复现

双卡H20， qwen3.5-9b，

Additional Information / 补充信息

No response

Start with the reported two-GPU H20 reproduction using Qwen3.5-9B and inspect the torch.distributed elastic traceback around the received death signal and SIGHUP handling. Determine what interrupts training at step 482 and document a reproducible fix or diagnostic, verified by completing the same training run without the interruption.