modelscope / modelscope/ms-swift
考虑支持 LitePPO or ‘reinforce++ baseline’
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 1.7k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 136
Description
Describe the feature
LitePPO：
1、鲁棒的优势归一化 (Robust Advantage Normalization): 该技术结合了组级别和批次级别的统计量。具体来说，优势的均值在组（group）级别计算，而标准差在整个批次（batch）级别计算。
2、token级损失聚合 (Token-Level Loss Aggregation): 与DAPO中采用的技术类似，LitePPO主张在计算总损失时，对批次内所有token的损失进行求和，然后除以token总数，而不是在序列之间取平均。
Paste any useful information
https://arxiv.org/abs/2508.08221
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the linked LitePPO paper and comparing its robust advantage normalization and token-level loss aggregation with the repository’s existing GRPO or PPO training paths. Done means implementing both requested behaviors with validation against the relevant training and loss calculations, but the issue names no files or tests to use.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100