modelscope / modelscope/ms-swift
qwen3-vl-30b的多机多卡megatron-swift grpo训练结束时报错异常退出
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 1.7k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 136
Description
所有steps都训练完成之后,异常退出了。这个是报错:traceback : Signal 6 (SIGABRT) received by PID 592
error_file:
exitcode : -6 (pid: 592)
rank : 19 (local_rank: 3)
host : cce-ig-jhi6i62h-s5pp6kok-k049m9py-03
time : 2025-12-02_13:14:06
[0]:
Root Cause (first observed failure):
Failures:
/mnt/volumes/ss-sai-bd-ga/liupengyi/ms-swift/swift/cli/_megatron/rlhf.py FAILED
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
raise ChildFailedError(
File "/usr/local/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 277, in launch_agent
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
return launch_agent(self._config, self._entrypoint, list(args))
File "/usr/local/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 143, in call
elastic_launch(
File "/usr/local/lib/python3.11/site-packages/torch/distributed/run.py", line 892, in run
run(args)
File "/usr/local/lib/python3.11/site-packages/torch/distributed/run.py", line 901, in main
^^^^^^^^^^^^^^^^^^
return f(*args, **kwargs)
File "/usr/local/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 357, in wrapper
main()
File "/us
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with swift/cli/_megatron/rlhf.py and the PyTorch distributed launcher context around the reported rank 19 SIGABRT after all training steps complete. Reproduce the multi-node, multi-GPU Qwen3-VL GRPO run if the missing configuration can be obtained; done means the completed training exits normally without ChildFailedError and the underlying failure is identified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100