dmlc / dmlc/dgl

Allow S3 file source for distributed training and partitioning

Open
#5,504 0 comments 0 reactions 1 assignee Claimed by @kylasa View on GitHub
feature request
Dominant language
Python
Stars
14.3k
Forks
3.1k
PR merge metrics
No merged PRs in 30d

Description

## 🚀 Feature
Currently, the distributed partitioning tools assume the existence of a shared filesystem in which all necessary files are available for each node in the clusters.

It would be easier for the user side if they were able to provide a source location on S3 and then individual workers downloaded the files they needed, instead of assuming the presence of a shared FS.

## Motivation

Setting up and maintaining a shared FS is an additional friction point for someone looking to try out distributed training in an enterprise setting.

## Alternatives

Our (GraphStorm) current solution is to download these files in our wrapper scripts, but this creates entanglement with DGL.

## Pitch

Add the ability for users to provide an S3 prefix under which data are arranged in a manner described in https://docs.dgl.ai/guide/distributed-preprocessing.html

Each individual worker then should download the files described in the metadata files as needed, instead of assuming a shared FS. This shouldn't replace the shared FS implementation, but add S3 as an additional option.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.