Lightning-AI / Lightning-AI/pytorch-lightning
Flatten the Strategy inheritance
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
## Proposed refactor
Flatten the Strategy inheritance:
Part of #10416
### Motivation
Reduce coupling between strategies, reduce unintentional overrides/inheritance and avoid silent failures
eg:
1. DeepSpeed inherited configure_ddp() from DDP, which is unnecessary and error-prone.
2. DDP and FSDP have totally different distributed behavior, but with current inheritance, adding a DDP strategy method will automatically enable this method for FSDP
### Pitch
Tasks list:
- FSDP inherit ParallelStrategy instead of DDP
- DeepSpeed inherit ParallelStrategy instead of DDP
- [RFC] DDPShard inherit ParallelStrategy instead of DDP
- [RFC] DDPShardSpawn inherit ParallelStrategy instead of DDPSpawn
- [RFC] TPUSpawn inherit ParallelStrategy instead of DDPSpawn
### Additional context
The downside of this task will be code duplications.
Align DDP/DDPSpawn precess group creation #11643 and Collective refactor #9414 will reduce the duplication.
______________________________________________________________________
#### If you enjoy Lightning, check out our other projects! ⚡
- [**Metrics**](https://github.com/PyTorchLightning/metrics): Machine learning metrics for distributed, scalable PyTorch applications.
- [**Lite**](https://pytorch-lightning.readthedocs.io/en/latest/starter/lightning_lite.html): enables pure PyTorch users to scale their existing code on any kind of device while retaining full control over their own loops and optimization logic.
- [**Flash**](https://github.com/PyTorchLightning/lightning-flash): The fastest way to get a Lightning baseline! A collection of tasks for fast prototyping, baselining, fine-tuning, and solving problems with deep learning.
- [**Bolts**](https://github.com/PyTorchLightning/lightning-bolts): Pretrained SOTA Deep Learning models, callbacks, and more for research and production with PyTorch Lightning and PyTorch.
- [**Lightning Transformers**](https://github.com/PyTorchLightning/lightning-transformers): Flexible interface for high-performance research using SOTA Transformers leveraging Pytorch Lightning, Transformers, and Hydra.
cc @justusschock @awaelchli @akihironitta @rohitgr7
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.