google / google/weather-tools

weather-mv: Improve tool's efficiency in terms of time & memory.

Open
#291 2 comments 0 reactions 0 assignees View on GitHub
enhancement P1 weather-mv
Dominant language
Python
Stars
252
Forks
60
Avg merge
8d 4h
Merged PRs (30d)
3

Description

### Time Efficient:
Make use of `gcloud alpha storage` in open_local() method, sinks.py.

Findings -- using `gsutil` for downloading the data from gcs to the local file system is 5 times slower compared to `gcloud alpha storage`.

### Memory Efficient:
Every time when we log `xr_dataset.nbytes` it will takes the complete dataset in-memory which is causing OOM killer invocation.
TODO: Find a better way for logging the dataset size.

### Real-time data ingestion into BQ:
beam.io.WriteToBigQuery() -- in case of batch pipeline data is not ingested into BQ in real-time. Because batch pipeline processes all elements before writing to BigQuery.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.