drivendataorg / drivendataorg/cloudpathlib

S3Path: support extremely fast bucket-to-bucket transfers

Open
#66 0 comments 0 reactions 0 assignees View on GitHub
caching cloudfile enhancement
Dominant language
Python
Stars
628
Forks
88
Avg merge
17h 28m
Merged PRs (30d)
2

Description

We've experimented with this tool for fast bucket-to-bucket transfers. It would be cool if we could leverage the logic here to make an incredibly fast bucket-to-bucket transfer:
https://github.com/generalui/s3p/

Here's a blog post with more deets:
https://medium.com/@shanebdavis/s3p-massively-parallel-s3-copying-9a9e466d0d74

I think the core pieces are:
- `async` for all of the requests
- use `prefix` and assumptions about list results in alphabetical order to split up the parallel "list" calls search space? (or something like that)
- get a massive number of requests goin' at once.

(This may also be out of scope for the package, just throwing it out there as useful for our purposes)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.