google / google/xarray-beam

Support opening datasets with file-like objects in a Beam pipeline

Open
#37 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
170
Forks
15
Avg merge
18h 27m
Merged PRs (30d)
1

Description

I experimented a bit more with this based on @mjwillson's suggestion.

Amazingly, it seems that uses file-like objects in Xarray _does_ actually work as used here, though making a local copy might still have better performance.

What _doesn't_ work yet -- but hopefully with small upstream changes to Xarray could work -- is passing xarray datasets opened with these file-like objects into a Beam pipeilne. That could let us do the actual data loading from netCDF in separate workers, which could be quite a win!

_Originally posted by @shoyer in https://github.com/google/xarray-beam/pull/31#discussion_r696246752_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.