Writeable backends via entrypoints
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
The backend refactor has gone a long way towards making it easier to implement custom backend readers via entry points. However, it is still not clear how to implement a writeable backend from a third party package as an entry point. Some of the reasons for this are:
- While our reading function (
open_dataset) has a generic name, our writing functions (Dataset.to_netcdf/Dataset.to_zarr) are still format specific. (Related to https://github.com/pydata/xarray/issues/3638). I propose we introduce a genericDataset.tomethod and deprecate the others. - The
BackendEntrypointbase class does not have a writing method, justopen_dataset:
https://github.com/pydata/xarray/blob/e0deb9cf0a5cd5c9e3db033fd13f075added9c1e/xarray/backends/common.py#L356-L370
(Related to https://github.com/pydata/xarray/issues/1970) - As a result, writing is implemented ad-hoc for each backend.
- This makes it impossible for a third-party package to to implement writing.
We should fix this situation! Here are the steps I would take.
- Decide on the desired API for writeable backends.
- Formalize this in the
BackendEntrypointbase class. - Refactor the existing writeable backends (netcdf4-python, h5netcdf, scipy, Zarr) to use this API
- Maybe deprecate
to_zarrandto_netcdf(or at least refactor to make a shallow call to a generic method) - Encourage third party implementors to try it (e.g. TileDB)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the BackendEntrypoint base class in xarray/backends/common.py and compare its open_dataset entry point with the existing Dataset.to_netcdf and Dataset.to_zarr paths. The issue is complete only after a writing API is decided, formalized, applied to the netcdf4-python, h5netcdf, scipy, and Zarr backends, and its third-party usability is considered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100