Create a performant, cloud-agnostic way to download & upload files to cloud buckets.
Open
- Dominant language
- Python
- Stars
- 252
- Forks
- 60
- Avg merge
- 8d 4h
- Merged PRs (30d)
- 3
Description
See discussion here: https://github.com/google/weather-tools/pull/254#discussion_r996220532
To investigate:
- Can we do better than `shutil.copyfileobj`?
- What are the optimal chunk sizes?
- Can we copy data in parallel?
- Are there optimizations we can do for large files?
One idea that @bahmandar has explored is calling gsutils in a subprocess (the CLI is really efficient at file transfer).
Contributor guide
Assessment
This issue has not been assessed yet.