modelscope / modelscope/ms-swift

多轮轨迹拆分场景中,`--num_generations` 与 `--generation_batch_size` 设置疑惑与改进讨论

Open
#7,849 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

question
Dominant language
Python
Stars
15.7k
Forks
1.7k
Avg merge
1d 16h
Merged PRs (30d)
136

Description

Checklist / 检查清单
  • I have searched existing issues, and this is a new question or discussion topic. / 我已经搜索过现有的 issues,确认这是一个新的问题与讨论。
Question Description / 问题描述

背景问题:在grpo Agent 训练场景中,我们想把一个完整的n轮轨迹拆分成多条子轨迹,将这些子轨迹与完整全部用于训练。但实践发现,实际用于模型参数更新的轨迹样本受--generation_batch_size限制。

参数设置:--num_generation 为4 ,--generation_batch_size 为4 ,--max_turns 为3

正文:按照如上参数设置,我们每次采样期望使用数据集中的1个问题,使用这个问题采样4条轨迹。假如这4条轨迹的对话轮次都是3,在拆分轨迹后,那我们期望有12条样本用于训练,然而实际上我们用于模型更新的样本只有--num_generation =4 条。下一次采样应该又是用新的问题了,而不是用剩下的拆分后的8条轨迹。

疑惑:这样会造成资源浪费,并没有把所有采样结果用于模型训练。当然好像也不太好设计,因为拆分后的轨迹数量随机,不一定能被num_device整除。想看看有没有比较好的方法把所有拆分后的数据都用于训练

版本:ms-swift 3.12.2

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the GRPO setup with --num_generation 4, --generation_batch_size 4, and --max_turns 3, then trace how split trajectories are queued for model updates. Review the batching and device-divisibility behavior described in the issue. Done means reaching a documented design or implementation path that uses eligible split samples without violating batch or device constraints.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.