google / google/xarray-beam

Simultaneously read multiple Datasets into an Xarray-Beam pipeline

Open
#68 2 comments 2 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
170
Forks
15
Avg merge
18h 27m
Merged PRs (30d)
1

Description

It is relatively common to need to load multiple xarray.Dataset objects, e.g., to compare two different models.

This currently can be done by loading data with separate calls to `xbeam.DatasetToChunks`, and by joining together the result with `beam.CoGroupBykey`. This works but is rather inefficient, involving an extra write of the data to disk. Ideally we could load the data in a single beam transform instead, e.g., `xbeam.DatasetToChunks([ds1, ds2], chunks)` would return a PCollection with elements of type `tuple[xbeam.Key, tuple[xarray.Dataset, xarray.Dataset]]`.

CC @alxmrs

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.