Simultaneously read multiple Datasets into an Xarray-Beam pipeline
Open
- Dominant language
- Python
- Stars
- 170
- Forks
- 15
- Avg merge
- 18h 27m
- Merged PRs (30d)
- 1
Description
It is relatively common to need to load multiple xarray.Dataset objects, e.g., to compare two different models.
This currently can be done by loading data with separate calls to `xbeam.DatasetToChunks`, and by joining together the result with `beam.CoGroupBykey`. This works but is rather inefficient, involving an extra write of the data to disk. Ideally we could load the data in a single beam transform instead, e.g., `xbeam.DatasetToChunks([ds1, ds2], chunks)` would return a PCollection with elements of type `tuple[xbeam.Key, tuple[xarray.Dataset, xarray.Dataset]]`.
CC @alxmrs
Contributor guide
Assessment
This issue has not been assessed yet.