Allow S3 file source for distributed training and partitioning
- Dominant language
- Python
- Stars
- 14.3k
- Forks
- 3.1k
- PR merge metrics
- No merged PRs in 30d
Description
## 🚀 Feature
Currently, the distributed partitioning tools assume the existence of a shared filesystem in which all necessary files are available for each node in the clusters.
It would be easier for the user side if they were able to provide a source location on S3 and then individual workers downloaded the files they needed, instead of assuming the presence of a shared FS.
## Motivation
Setting up and maintaining a shared FS is an additional friction point for someone looking to try out distributed training in an enterprise setting.
## Alternatives
Our (GraphStorm) current solution is to download these files in our wrapper scripts, but this creates entanglement with DGL.
## Pitch
Add the ability for users to provide an S3 prefix under which data are arranged in a manner described in https://docs.dgl.ai/guide/distributed-preprocessing.html
Each individual worker then should download the files described in the metadata files as needed, instead of assuming a shared FS. This shouldn't replace the shared FS implementation, but add S3 as an additional option.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.