pydata / pydata/xarray

Expose "memory" argument to the "netcdf4" engine

Open
#6,955 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

Is your feature request related to a problem?

As of today, xarray can't open a netcdf residing in a memory buffer, which is possible in netCDF4 library through the memory argument of the netCDF4.Dataset class constructor.

Describe the solution you'd like

It would be nice to create an xarray.Dataset from a netcdf file in memory like so:

buf: bytes = ...
ds = xr.open_dataset("dummy.nc", engine="netcdf4", memory=buf)

But this fails with

File ~/.asdf/installs/python/3.9.13/lib/python3.9/site-packages/xarray/backends/api.py:531, in open_dataset(filename_or_obj, engine, chunks, cache, decode_cf, mask_and_scale, decode_times, decode_timedelta, use_cftime, concat_characters, decode_coords, drop_variables, inline_array, backend_kwargs, **kwargs)
    519 decoders = _resolve_decoders_kwargs(
    520     decode_cf,
    521     open_backend_dataset_parameters=backend.open_dataset_parameters,
   (...)
    527     decode_coords=decode_coords,
    528 )
    530 overwrite_encoded_chunks = kwargs.pop("overwrite_encoded_chunks", None)
--> 531 backend_ds = backend.open_dataset(
    532     filename_or_obj,
    533     drop_variables=drop_variables,
    534     **decoders,
    535     **kwargs,
    536 )
    537 ds = _dataset_from_backend_dataset(
    538     backend_ds,
    539     filename_or_obj,
   (...)
    547     **kwargs,
    548 )
    549 return ds

TypeError: open_dataset() got an unexpected keyword argument 'memory'
Describe alternatives you've considered

An alternative is to save the buffer into a temporary file and open it the usual way, but it feels cumbersome when there is already an argument to open directly from memory.

Additional context

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at xarray.open_dataset and trace how the netcdf4 engine forwards arguments to netCDF4.Dataset. Reproduce the reported TypeError with an in-memory bytes buffer, then verify that the example using engine="netcdf4" and memory=buf opens the dataset successfully. Add or update coverage for this behavior if the relevant test location is identified.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.