`to_netcdf` fails to write to files previously opened with `open_mfdataset`
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
What happened?
I've been working in a system where I occasionally read a multifile dataset and, in effect within the same Python process, subsequently want to write to some of the files containing that dataset. Instead, I get permission errors when I try to write to the file.
What did you expect to happen?
I expected to not get a permission error.
Minimal Complete Verifiable Example
import xarray as xr
ds = xr.Dataset(
attrs={'blah': -1},
coords=({'x': range(10)}),
data_vars={'y': ('x', [10*i for i in range(10)])},
)
ds.isel(x=slice(0,5)).to_netcdf('file1.nc')
ds.isel(x=slice(5,10)).to_netcdf('file2.nc')
a = xr.open_mfdataset(paths=['file1.nc', 'file2.nc'])
ds.to_netcdf('file1.nc')
MVCE confirmation
- Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
- Complete example — the example is self-contained, including all data and the text of any traceback.
- Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
- New issue — a search of GitHub Issues suggests this is not a duplicate.
- Recent environment — the issue occurs with the latest version of xarray and its dependencies.
Relevant log output
Traceback (most recent call last):
File "/home/wball/.local/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 211, in _acquire_with_cache_info
file = self._cache[self._key]
~~~~~~~~~~~^^^^^^^^^^^
File "/home/wball/.local/lib/python3.12/site-packages/xarray/backends/lru_cache.py", line 56, in __getitem__
value = self._cache[key]
~~~~~~~~~~~^^^^^
KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('/home/wball/tmp/file1.nc',), 'a', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False)), '11d144ce-327c-44e3-82f5-fb97c2d66e65']
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/wball/tmp/xarray-mwe.py", line 14, in <module>
ds.to_netcdf('file1.nc')
File "/home/wball/.local/lib/python3.12/site-packages/xarray/core/dataset.py", line 2329, in to_netcdf
return to_netcdf( # type: ignore # mypy cannot resolve the overloads:(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wball/.local/lib/python3.12/site-packages/xarray/backends/api.py", line 1343, in to_netcdf
store = store_open(target, mode, format, group, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wball/.local/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 408, in open
return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wball/.local/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 355, in __init__
self.format = self.ds.data_model
^^^^^^^
File "/home/wball/.local/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 417, in ds
return self._acquire()
^^^^^^^^^^^^^^^
File "/home/wball/.local/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 411, in _acquire
with self._manager.acquire_context(needs_lock) as root:
File "/usr/lib64/python3.12/contextlib.py", line 137, in __enter__
return next(self.gen)
^^^^^^^^^^^^^^
File "/home/wball/.local/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 199, in acquire_context
file, cached = self._acquire_with_cache_info(needs_lock)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wball/.local/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 217, in _acquire_with_cache_info
file = self._opener(*self._args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "src/netCDF4/_netCDF4.pyx", line 2469, in netCDF4._netCDF4.Dataset.__init__
File "src/netCDF4/_netCDF4.pyx", line 2028, in netCDF4._netCDF4._ensure_nc_success
PermissionError: [Errno 13] Permission denied: '/home/wball/tmp/file1.nc'
Anything else we need to know?
No response
Environment
INSTALLED VERSIONS
commit: None
python: 3.12.4 (main, Jun 7 2024, 00:00:00) [GCC 13.3.1 20240522 (Red Hat 13.3.1-1)]
python-bits: 64
OS: Linux
OS-release: 6.9.12-100.fc39.x86_64
machine: x86_64
processor:
byteorder: little
LC_ALL: None
LANG: en_GB.UTF-8
LOCALE: ('en_GB', 'UTF-8')
libhdf5: 1.12.2
libnetcdf: 4.9.3-development
xarray: 2024.7.0
pandas: 2.2.2
numpy: 1.26.3
scipy: 1.11.1
netCDF4: 1.6.5
pydap: None
h5netcdf: None
h5py: 3.11.0
zarr: None
cftime: 1.6.3
nc_time_axis: None
iris: None
bottleneck: 1.3.7
dask: 2024.1.0
distributed: None
matplotlib: 3.8.4
cartopy: 0.23.0
seaborn: None
numbagg: None
fsspec: 2024.6.0
cupy: None
pint: None
sparse: None
flox: None
numpy_groupies: None
setuptools: 67.7.2
pip: 23.2.1
conda: None
pytest: 7.4.4
mypy: None
IPython: 8.14.0
sphinx: 7.2.6
None
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the example and reading xarray/backends/file_manager.py and xarray/backends/netCDF4_.py, the paths shown in the traceback. Trace how open_mfdataset retains file handles before the later to_netcdf call. Done means the example can reopen file1.nc for writing without a permission error, with regression coverage for this sequence.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100