Expose "memory" argument to the "netcdf4" engine
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
Is your feature request related to a problem?
As of today, xarray can't open a netcdf residing in a memory buffer, which is possible in netCDF4 library through the memory argument of the netCDF4.Dataset class constructor.
Describe the solution you'd like
It would be nice to create an xarray.Dataset from a netcdf file in memory like so:
buf: bytes = ...
ds = xr.open_dataset("dummy.nc", engine="netcdf4", memory=buf)
But this fails with
File ~/.asdf/installs/python/3.9.13/lib/python3.9/site-packages/xarray/backends/api.py:531, in open_dataset(filename_or_obj, engine, chunks, cache, decode_cf, mask_and_scale, decode_times, decode_timedelta, use_cftime, concat_characters, decode_coords, drop_variables, inline_array, backend_kwargs, **kwargs)
519 decoders = _resolve_decoders_kwargs(
520 decode_cf,
521 open_backend_dataset_parameters=backend.open_dataset_parameters,
(...)
527 decode_coords=decode_coords,
528 )
530 overwrite_encoded_chunks = kwargs.pop("overwrite_encoded_chunks", None)
--> 531 backend_ds = backend.open_dataset(
532 filename_or_obj,
533 drop_variables=drop_variables,
534 **decoders,
535 **kwargs,
536 )
537 ds = _dataset_from_backend_dataset(
538 backend_ds,
539 filename_or_obj,
(...)
547 **kwargs,
548 )
549 return ds
TypeError: open_dataset() got an unexpected keyword argument 'memory'
Describe alternatives you've considered
An alternative is to save the buffer into a temporary file and open it the usual way, but it feels cumbersome when there is already an argument to open directly from memory.
Additional context
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at xarray.open_dataset and trace how the netcdf4 engine forwards arguments to netCDF4.Dataset. Reproduce the reported TypeError with an in-memory bytes buffer, then verify that the example using engine="netcdf4" and memory=buf opens the dataset successfully. Add or update coverage for this behavior if the relevant test location is identified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100