Signal 9 (SIGKILL) received by PID 394968

`
export WANDB_MODE=offline
export WANDB_DIR=/lyx/qwenvl-rl/wandb_local
#export NCCL_IB_DISABLE=1
#export NCCL_P2P_DISABLE=1
model_path=/lyx/model/Qwen3-VL-4B-Thinking

#--gradient_checkpointing true
IMAGE_MAX_TOKEN_NUM=1024
PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True'
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5
NPROC_PER_NODE=6
swift rlhf
--rlhf_type grpo
--model $model_path
--external_plugins reward.py
--reward_funcs name_acc format repetition
--repetition_n_grams 5
--use_vllm true
--vllm_mode server
--vllm_server_host 127.0.0.1
--vllm_server_port 8001
--train_type full
--torch_dtype bfloat16
--dataset /lyx/qwenvl-rl/swift/train0107.jsonl
--load_from_cache_file true
--max_completion_length 3000
--num_train_epochs 3
--per_device_train_batch_size 4
--per_device_eval_batch_size 4
--learning_rate 1e-6
--gradient_accumulation_steps 2
--save_strategy 'steps'
--eval_strategy 'steps'
--eval_steps 75
--save_steps 75
--save_total_limit 10
--logging_steps 1
--output_dir /lyx/qwenvl-rl/swift/output/GRPO_0107
--warmup_ratio 0.01
--dataloader_num_workers 1
--num_generations 8
--temperature 1.0
--log_completions true
--report_to wandb
--num_iterations 1
--deepspeed zero2
--async_generate false
--beta 0.001 `

`{'loss': 6.853e-05, 'grad_norm': 0.19923472, 'learning_rate': 4.7e-07, 'completions/mean_length': 719.10418701, 'completions/min_length': 332.0, 'completions/max_length': 3000.0, 'completions/clipped_ratio': 0.02083333, 'reward': 1.89254951, 'reward_std': 0.09405056, 'frac_reward_zero_std': 0.66666669, 'rewards/SignboardORM/mean': 0.91437501, 'rewards/SignboardORM/std': 0.14451541, 'rewards/Format/mean': 0.97916669, 'rewards/Format/std': 0.14433756, 'rewards/RepetitionPenalty/mean': -0.00099206, 'rewards/RepetitionPenalty/std': 0.00687322, 'kl': 0.06647041, 'clip_ratio/low_mean': 0.0, 'clip_ratio/low_min': 0.0, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.0, 'epoch': 1.58, 'global_step/max_steps': '79/150', 'percentage': '52.67%', 'elapsed_time': '4h 2m 1s', 'remaining_time': '3h 37m 30s', 'memory(GiB)': 50.74, 'train_speed(iter/s)': 0.00544}
{'loss': 6.241e-05, 'grad_norm': 0.27866709, 'learning_rate': 4.6e-07, 'completions/mean_length': 756.27087402, 'completions/min_length': 395.0, 'completions/max_length': 2007.0, 'completions/clipped_ratio': 0.0, 'reward': 1.86312509, 'reward_std': 0.04772971, 'frac_reward_zero_std': 0.83333337, 'rewards/SignboardORM/mean': 0.86312503, 'rewards/SignboardORM/std': 0.24540159, 'rewards/Format/mean': 1.0, 'rewards/Format/std': 0.0, 'rewards/RepetitionPenalty/mean': 0.0, 'rewards/RepetitionPenalty/std': 0.0, 'kl': 0.06031546, 'clip_ratio/low_mean': 0.0, 'clip_ratio/low_min': 0.0, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.0, 'epoch': 1.6, 'global_step/max_steps': '80/150', 'percentage': '53.33%', 'elapsed_time': '4h 4m 36s', 'remaining_time': '3h 34m 1s', 'memory(GiB)': 50.74, 'train_speed(iter/s)': 0.005451}
Train: 53%|██████████████████████████████████████████████▍ | 80/150 [4:04:36<2:57:40, 152.29s/it]W0112 06:26:40.654000 394898 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 394963 closing signal SIGTERM
W0112 06:26:40.655000 394898 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 394964 closing signal SIGTERM
W0112 06:26:40.657000 394898 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 394965 closing signal SIGTERM
W0112 06:26:40.659000 394898 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 394966 closing signal SIGTERM
W0112 06:26:40.660000 394898 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 394967 closing signal SIGTERM
E0112 06:26:42.644000 394898 torch/distributed/elastic/multiprocessing/api.py:874] failed (exitcode: -9) local_rank: 5 (pid: 394968) of binary: /usr/bin/python
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 905, in
main()
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 357, in wrapper
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 901, in main
run(args)
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 892, in run
elastic_launch(
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 143, in call
return launch_agent(self._config, self._entrypoint, list(args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 277, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

/usr/local/lib/python3.12/dist-packages/swift/cli/rlhf.py FAILED

Failures:
<NO_OTHER_FAILURES>

Root Cause (first observed failure):
[0]:
time : 2026-01-12_06:26:40
host : vlm-with-h20-nanjing-1.vlm-with-h20-nanjing.ns-prjqg82p-1575531-production.svc.cluster.local
rank : 5 (local_rank: 5)
exitcode : -9 (pid: 394968)
error_file: <N/A>
traceback : Signal 9 (SIGKILL) received by PID 394968
============================================================`

Start by reproducing the reported swift rlhf command and reading the failure path through swift/cli/rlhf.py and torch.distributed.run. Compare the rank-5 SIGKILL with the preceding distributed-launcher logs and establish the cause and a reproducible resolution for the training run.