ByteDance-Seed / ByteDance-Seed/Bagel
训练脚本中使用的FDSP框架是不是不支持在单卡上训练?
- Dominant language
- Python
- Stars
- 6.2k
- Forks
- 545
- PR merge metrics
- No merged PRs in 30d
Description
我在尝试冻结绝大部分参数以加载到一张A800上进行训练时总是报错:
```
E0604 16:32:11.373000 19000 site-packages/torch/distributed/elastic/multiprocessing/api.py:869] failed (exitcode: -9) local_rank: 0 (pid: 19025) of binary: /root/miniconda3/envs/bagel/bin/python3.10
Traceback (most recent call last):
File "/root/miniconda3/envs/bagel/bin/torchrun", line 8, in
sys.exit(main())
File "/root/miniconda3/envs/bagel/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
return f(*args, **kwargs)
File "/root/miniconda3/envs/bagel/lib/python3.10/site-packages/torch/distributed/run.py", line 919, in main
run(args)
File "/root/miniconda3/envs/bagel/lib/python3.10/site-packages/torch/distributed/run.py", line 910, in run
elastic_launch(
File "/root/miniconda3/envs/bagel/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 138, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/root/miniconda3/envs/bagel/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 269, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
======================================================
train/pretrain_unified_navit.py FAILED
------------------------------------------------------
Failures:
------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2025-06-04_16:32:11
host : autodl-container-b8384da009-75e4d3df
rank : 0 (local_rank: 0)
exitcode : -9 (pid: 19025)
error_file:
traceback : Signal 9 (SIGKILL) received by PID 19025
======================================================
```
我不知道这是不是因为FDSP框架的问题
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with train/pretrain_unified_navit.py and the torchrun command, then reproduce the frozen-parameter training setup on a single A800. Investigate whether FDSP supports this configuration and what causes the reported SIGKILL; document the confirmed limitation or root cause and the conditions for a successful run.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100