Lightning-AI / Lightning-AI/pytorch-lightning
Deprecate `prepare_data_per_node` and auto detect shared file-system
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
## Proposed refactor
This was proposed by @tchaton on live-stream. `prepare_data_per_node`can be identified by dumping a file on rank 0 and checking whether all other ranks can see that file or not. In the case of a multi-node system, if the file system is shared then all of them will be able to see that file, else not.
### Motivation
This way users won't have to take care of setting this flag manually.
### Pitch
possible pseudocode (feel free to update this or comment)
```py
if global_rank == 0:
filename = 'something_{uuid()}.txt'
broadcast(filename)
dump_file(filename)
barrier()
if global_rank == 0
is_file_present = does_file_exist(filename)
is_file_present = all_gather(is_file_present)
prepare_data_per_node = is_file_present.all()
prepare_data_per_node = broadcast(prepare_data_per_node)
```
any other alternatives??
### Additional context
______________________________________________________________________
#### If you enjoy Lightning, check out our other projects! ⚡
- [**Metrics**](https://github.com/PyTorchLightning/metrics): Machine learning metrics for distributed, scalable PyTorch applications.
- [**Lite**](https://pytorch-lightning.readthedocs.io/en/latest/starter/lightning_lite.html): enables pure PyTorch users to scale their existing code on any kind of device while retaining full control over their own loops and optimization logic.
- [**Flash**](https://github.com/PyTorchLightning/lightning-flash): The fastest way to get a Lightning baseline! A collection of tasks for fast prototyping, baselining, fine-tuning, and solving problems with deep learning.
- [**Bolts**](https://github.com/PyTorchLightning/lightning-bolts): Pretrained SOTA Deep Learning models, callbacks, and more for research and production with PyTorch Lightning and PyTorch.
- [**Lightning Transformers**](https://github.com/PyTorchLightning/lightning-transformers): Flexible interface for high-performance research using SOTA Transformers leveraging Pytorch Lightning, Transformers, and Hydra.
cc @borda @justusschock @kaushikb11 @awaelchli @ananthsub @ninginthecloud @jjenniferdai @rohitgr7 @akihironitta
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the prepare_data_per_node option and the distributed data-preparation entry points. Review the proposed rank, barrier, file-visibility, and broadcast flow, along with existing distributed tests if available. Done means the flag's deprecation and automatic shared-filesystem detection have agreed behavior and coverage across shared and non-shared filesystems.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100