Pipeline Parallelism + FSDP
Open
@H-Huang is already working on this.
Since Oct 27, 2025.
enhancement
module: pipelining
question
- Dominant language
- Python
- Stars
- 5.8k
- Forks
- 997
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 230
Description
On PP + FSDP and PP + TP + FSDP:
- Is there any documentation on how these different parallelisms compose?
- What are the largest training runs these strategies have been tested on?
- Are there benchmarks for how these strategies compare against other distributed training frameworks that expose similar parallelisms?
Particularly interested in how PP + FSDP work together as it seems DeepSpeed explicitly disallows ZeRO 2/3 + PP (see here specifically, and here for discussion).
@wconstab @weifengpy @wanchaol
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.