Expose xarray's h5py serialization capabilites as public API?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
Xarray has a magic ability to serialize h5py datasets. We should expose this somehow and allow it to be used outside of xarray.
Consider the following example:
import s3fs
import h5py
import dask.array as dsa
import xarray as xr
import cloudpickle
url = 'noaa-goes16/ABI-L2-RRQPEF/2020/001/00/OR_ABI-L2-RRQPEF-M6_G16_s20200010000216_e20200010009524_c20200010010034.nc'
fs = s3fs.S3FileSystem(anon=True)
f = fs.open(url)
ds = h5py.File(f, mode='r')
data = dsa.from_array(ds['RRQPE'])
_ = cloudpickle.dumps(data)
This raises TypeError: h5py objects cannot be pickled.
However, if I read the file with xarray...
ds = xr.open_dataset(f, chunks={})
data = ds['RRQPE'].data
_ = cloudpickle.dumps(data)
It works just fine. This has come up in several places (e.g. https://github.com/dask/s3fs/issues/337, https://github.com/dask/distributed/issues/2787).
It seems like the ability to pickle these arrays is broadly useful, beyond xarray.
- How does our magic work?
- What would it look like to break this magic out and expose it as public API (or inside another package)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the h5py pickling failure in the issue examples and tracing how xarray makes the equivalent dask-backed array serializable. Determine where that behavior currently lives, then assess a public API or separate-package design and define the required tests and documentation before implementation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100