huggingface / huggingface/accelerate
Feature request: FSDP for TPUs
Open
feature request
TPU
- Dominant language
- Python
- Stars
- 9.9k
- Forks
- 1.5k
- Avg merge
- 5d 2h
- Merged PRs (30d)
- 27
Description
A recent contribution to the pytorch_xla repo allows using FSDP in PyTorch XLA for sharding Module parameters across data-parallel workers. https://github.com/pytorch/xla/pull/3431
Some motivation behind this: It may be possible perform inference with OPT 30B on Google Colab without needing a Pro subscription, which I think many people will appreciate.
What will be needed to add it to accelerate?
Contributor guide
Assessment
This issue has not been assessed yet.