[训练耗时问题] 基于strands-agents魔改的generate训练耗时特别慢
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8.5k
- Forks
- 1.3k
- Avg merge
- 5h 36m
- Merged PRs (30d)
- 22
Description
基于example中的strands-agent,魔改并替换上自己的agent框架,训练集用dapo-math-17k.jsonl,模型用qwen3-30B,8卡,发现训练一个step需要2小时,这个耗时是合理的吗?我理解训练集中所有样本的rollout应该是集中起来一个list然后异步执行的吧,而且工具只定义了一个python code,轮次也没那么多,不应该这么久才对。不知道有经验的大佬们能否告知下是否是我魔改的问题,还是说这个训练速度是正常的。
另外我发现rollout_id==0的eval的log,需要运行很久才出来,理论上不应该很早就执行这段代码了吗?
# train loop.
# note that for async training, one can change the position of the sync operation(ray.get).
for rollout_id in range(args.start_rollout_id, args.num_rollout):
# TODO extract the duplicated eval logic
if args.eval_interval is not None and rollout_id == 0:
ray.get(rollout_manager.eval.remote(rollout_id))
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the example strands-agent integration and the training loop around rollout_manager.eval.remote(rollout_id) and ray.get. Reproduce the reported timing with dapo-math-17k.jsonl, Qwen3-30B, and 8 cards, then trace rollout scheduling and the rollout_id==0 evaluation path. Done means identifying whether the delay is caused by the customization or expected execution behavior, with evidence from timings or logs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100