drivendataorg / drivendataorg/cloudpathlib
S3Path: support extremely fast bucket-to-bucket transfers
- Dominant language
- Python
- Stars
- 628
- Forks
- 88
- Avg merge
- 17h 28m
- Merged PRs (30d)
- 2
Description
We've experimented with this tool for fast bucket-to-bucket transfers. It would be cool if we could leverage the logic here to make an incredibly fast bucket-to-bucket transfer:
https://github.com/generalui/s3p/
Here's a blog post with more deets:
https://medium.com/@shanebdavis/s3p-massively-parallel-s3-copying-9a9e466d0d74
I think the core pieces are:
- `async` for all of the requests
- use `prefix` and assumptions about list results in alphabetical order to split up the parallel "list" calls search space? (or something like that)
- get a massive number of requests goin' at once.
(This may also be out of scope for the package, just throwing it out there as useful for our purposes)
Contributor guide
Assessment
This issue has not been assessed yet.