Add support for direct S3 access on SageMaker tasks
- Dominant language
- Python
- Stars
- 452
- Forks
- 76
- PR merge metrics
- No merged PRs in 30d
Description
Because DistDGL and by extension GraphStorm has an assumption of a shared filesystem to function properly, in our SageMaker implementations need to implement various downloads and uploads to "fake" the existence of a shared filesystem, by downloading data locally to specific locations per instance.
This introduces a maintenance burden as we can't make the same environment assumptions for our SageMaker vs. EC2 with EFS execution, and introduces a lot of [glue code](https://proceedings.neurips.cc/paper_files/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf), to make the two system compatible.
[Mountpoint for S3](https://github.com/awslabs/mountpoint-s3) is an AWS project that allows entire S3 buckets to mounted onto EC2 instances and treated a (mostly) regular filesystem. If we are able to use S3 buckets as virtual shared filesystems for SageMaker we should be able to simplify and align the codebase. We note the use-cases suggested by the mountpoint-s3 project align with ours:
> Mountpoint for Amazon S3 is optimized for applications that need high read throughput to large objects, potentially from many clients at once, and to write new objects sequentially from a single client at a time. This means it's a great fit for applications that use a file interface to:
> * read large objects from S3, potentially from many instances concurrently, without downloading them to local storage first
> * access only some S3 objects out of a larger data set, but can't predict which objects in advance
> * upload their output to S3 directly, or upload files from local storage with tools like cp
>
> but probably not the right fit for applications that:
> * use file operations that S3 doesn't natively support, like directory renaming or symlinks
> * ( make edits to existing files (don't work on your Git repository or run vim in Mountpoint 😄)
We propose starting with a POC that modifies our SageMaker images and entry points to use mountpoint-s3, _but does not affect the user-facing launch scripts_, providing a backwards-compatible solution for our users.
Our first target will be adding GraphBolt support to SageMaker DistPartition, which is currently not possible, because DistDGL to GraphBolt partition conversion assumes that the leader instance has access to the entire distributed graph on disk. Following that, we can migrate our other SageMaker tasks to mountpoint-s3, where shared filesystems are normally required:
- [ ] DistPartition, remove download/upload of data from S3
- [ ] DistTraining
- [ ] DistInference
Contributor guide
Research direction
Start by locating the SageMaker images and entry points, then inspect the DistPartition implementation and its current S3 download/upload flow. The proof of concept is done when GraphBolt support can use Mountpoint for S3 without changing user-facing launch scripts; later migration of DistTraining and DistInference is also listed as follow-up work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud, distributed-systems, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100