google / google/weather-tools

Create a performant, cloud-agnostic way to download & upload files to cloud buckets.

Open
#256 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
252
Forks
60
Avg merge
8d 4h
Merged PRs (30d)
3

Description

See discussion here: https://github.com/google/weather-tools/pull/254#discussion_r996220532

To investigate:
- Can we do better than `shutil.copyfileobj`?
- What are the optimal chunk sizes?
- Can we copy data in parallel?
- Are there optimizations we can do for large files?

One idea that @bahmandar has explored is calling gsutils in a subprocess (the CLI is really efficient at file transfer).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.