ByteDance-Seed / ByteDance-Seed/Bagel

训练脚本中使用的FDSP框架是不是不支持在单卡上训练?

Open
#117 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
6.2k
Forks
545
PR merge metrics
No merged PRs in 30d

Description

我在尝试冻结绝大部分参数以加载到一张A800上进行训练时总是报错:
```

E0604 16:32:11.373000 19000 site-packages/torch/distributed/elastic/multiprocessing/api.py:869] failed (exitcode: -9) local_rank: 0 (pid: 19025) of binary: /root/miniconda3/envs/bagel/bin/python3.10
Traceback (most recent call last):
File "/root/miniconda3/envs/bagel/bin/torchrun", line 8, in
sys.exit(main())
File "/root/miniconda3/envs/bagel/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
return f(*args, **kwargs)
File "/root/miniconda3/envs/bagel/lib/python3.10/site-packages/torch/distributed/run.py", line 919, in main
run(args)
File "/root/miniconda3/envs/bagel/lib/python3.10/site-packages/torch/distributed/run.py", line 910, in run
elastic_launch(
File "/root/miniconda3/envs/bagel/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 138, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/root/miniconda3/envs/bagel/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 269, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
======================================================
train/pretrain_unified_navit.py FAILED
------------------------------------------------------
Failures:

------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2025-06-04_16:32:11
host : autodl-container-b8384da009-75e4d3df
rank : 0 (local_rank: 0)
exitcode : -9 (pid: 19025)
error_file:
traceback : Signal 9 (SIGKILL) received by PID 19025
======================================================
```
我不知道这是不是因为FDSP框架的问题

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with train/pretrain_unified_navit.py and the torchrun command, then reproduce the frozen-parameter training setup on a single A800. Investigate whether FDSP supports this configuration and what causes the reported SIGKILL; document the confirmed limitation or root cause and the conditions for a successful run.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.