Support opening datasets with file-like objects in a Beam pipeline
Open
- Dominant language
- Python
- Stars
- 170
- Forks
- 15
- Avg merge
- 18h 27m
- Merged PRs (30d)
- 1
Description
I experimented a bit more with this based on @mjwillson's suggestion.
Amazingly, it seems that uses file-like objects in Xarray _does_ actually work as used here, though making a local copy might still have better performance.
What _doesn't_ work yet -- but hopefully with small upstream changes to Xarray could work -- is passing xarray datasets opened with these file-like objects into a Beam pipeilne. That could let us do the actual data loading from netCDF in separate workers, which could be quite a win!
_Originally posted by @shoyer in https://github.com/google/xarray-beam/pull/31#discussion_r696246752_
Contributor guide
Assessment
This issue has not been assessed yet.