integrate MPIFileUtils with S3 backends
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 200
- Forks
- 85
- Avg merge
- 3d 21h
- Merged PRs (30d)
- 2
Description
This is firmly in the "wishlist" camp so far, mainly just seeing if there is interest behind this.
It would be useful for MPIFileUtils commands to be able to read/write/list from/to S3 buckets for migration between archive and online storage. This would allow scalable parallel data movement for large datasets.
Some restrictions would be needed on how the S3 backend was accessed, for example only migrating a single object/file with a single task, or chunking large files into multi-part uploads.
The POSIX attribute save/restore should follow existing conventions used by AWS and others on how they are stored in tags on the objects, so that import/export can preserve the file attributes across a round-trip.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names MPIFileUtils commands but no files or tests; start by locating their existing storage and POSIX attribute handling, then review AWS S3 object-tag conventions. Done would require an agreed design and implementation for S3 read, write, list, parallel migration, multipart uploads, and attribute round-tripping.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, c
- Domain
- cloud, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100