Error when training MoE with PPO
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8.5k
- Forks
- 1.3k
- Avg merge
- 5h 36m
- Merged PRs (30d)
- 22
Description
Hi, thanks for this great project. I got an error when training MoE (qwen30b-a3b) using PPO. I have tried GRPO, and it was all good. Now I changed --advantage-estimator to ppo, but got an error:
It seems that the advantage is None. How can I fix this problem? And it would also help if you could provide an example script for using PPO, since currently most examples are using GRPO. Thank you!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing MoE training with qwen30b-a3b and --advantage-estimator ppo, then compare the failing path with the working GRPO configuration. Trace where the advantage becomes None and document the required PPO setup; done means PPO training works and an example script is available.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100