Lightning-AI / Lightning-AI/pytorch-lightning

Deprecate `prepare_data_per_node` and auto detect shared file-system

Open
#12,159 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature trainer
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

## Proposed refactor

This was proposed by @tchaton on live-stream. `prepare_data_per_node`can be identified by dumping a file on rank 0 and checking whether all other ranks can see that file or not. In the case of a multi-node system, if the file system is shared then all of them will be able to see that file, else not.

### Motivation

This way users won't have to take care of setting this flag manually.

### Pitch

possible pseudocode (feel free to update this or comment)
```py
if global_rank == 0:
filename = 'something_{uuid()}.txt'
broadcast(filename)
dump_file(filename)

barrier()

if global_rank == 0
is_file_present = does_file_exist(filename)
is_file_present = all_gather(is_file_present)
prepare_data_per_node = is_file_present.all()
prepare_data_per_node = broadcast(prepare_data_per_node)
```

any other alternatives??

### Additional context

______________________________________________________________________

#### If you enjoy Lightning, check out our other projects! ⚡

- [**Metrics**](https://github.com/PyTorchLightning/metrics): Machine learning metrics for distributed, scalable PyTorch applications.

- [**Lite**](https://pytorch-lightning.readthedocs.io/en/latest/starter/lightning_lite.html): enables pure PyTorch users to scale their existing code on any kind of device while retaining full control over their own loops and optimization logic.

- [**Flash**](https://github.com/PyTorchLightning/lightning-flash): The fastest way to get a Lightning baseline! A collection of tasks for fast prototyping, baselining, fine-tuning, and solving problems with deep learning.

- [**Bolts**](https://github.com/PyTorchLightning/lightning-bolts): Pretrained SOTA Deep Learning models, callbacks, and more for research and production with PyTorch Lightning and PyTorch.

- [**Lightning Transformers**](https://github.com/PyTorchLightning/lightning-transformers): Flexible interface for high-performance research using SOTA Transformers leveraging Pytorch Lightning, Transformers, and Hydra.

cc @borda @justusschock @kaushikb11 @awaelchli @ananthsub @ninginthecloud @jjenniferdai @rohitgr7 @akihironitta

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the prepare_data_per_node option and the distributed data-preparation entry points. Review the proposed rank, barrier, file-visibility, and broadcast flow, along with existing distributed tests if available. Done means the flag's deprecation and automatic shared-filesystem detection have agreed behavior and coverage across shared and non-shared filesystems.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
distributed-systems
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.