to_netcdf() corrupts data when overwriting locked files in Jupyter

What happened?

to_netcdf() exhibits dangerous behavior when trying to overwrite a NetCDF file that was previously opened in another Jupyter notebook. The method:

1. Throws a PermissionError (expected)
2. BUT simultaneously truncates the target file to 0 bytes [DATA IS LOST]
3. File locks persist even after closing the notebook that opened the file
4. Attempting to open the corrupted 0-byte file gives a misleading error about missing IO backends

This results in silent data corruption. User loses their data even though the operation "failed."

What did you expect to happen?

to_netcdf() should either:

1. Succeed completely, or
2. Fail cleanly without modifying the existing file

Currently, we have a partial failure that erases existing data (unacceptable).

Minimal Complete Verifiable Example

# **Note: This bug requires a multi-notebook Jupyter environment to reproduce.**

# Step 1: Create and save dataset in Notebook A
import xarray as xr
import numpy as np
ds = xr.Dataset({'temp': (('x', 'y'), np.random.rand(10, 10))})
ds.to_netcdf('test.nc')  # Works fine

# Step 2: Open dataset in Notebook B  
import xarray as xr
data = xr.open_dataset('test.nc')  # Creates file lock

# Step 3: Close Notebook B (lock persists!)

# Step 4: Back in Notebook A, try to overwrite
import xarray as xr
import numpy as np
ds2 = xr.Dataset({'temp': (('x', 'y'), np.random.rand(5, 5))})
ds2.to_netcdf('test.nc')  
# Result: PermissionError BUT file is truncated to 0 bytes! [DATA LOST]

# Step 5: Try to read the corrupted file
corrupted = xr.open_dataset('test.nc')
# Result: Misleading error about missing IO backends

MVCE confirmation

• Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• Complete example — the example is self-contained, including all data and the text of any traceback.
• Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• New issue — a search of GitHub Issues suggests this is not a duplicate.
• Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
File c:\user\path\Miniconda3\envs\env\Lib\site-packages\xarray\backends\file_manager.py:211, in CachingFileManager._acquire_with_cache_info(self, needs_lock)
    210 try:
--> 211     file = self._cache[self._key]
    212 except KeyError:

File c:\user\path\Miniconda3\envs\env\Lib\site-packages\xarray\backends\lru_cache.py:56, in LRUCache.__getitem__(self, key)
     55 with self._lock:
---> 56     value = self._cache[key]
     57     self._cache.move_to_end(key)

KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('c:\\user\\path\\test.nc',), 'a', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False)), 'd5f70a75-468a-4959-afad-e326412ec7ab']

During handling of the above exception, another exception occurred:

PermissionError                           Traceback (most recent call last)
Cell In[9], line 1
----> 1 test_xr.to_netcdf('test.nc')

File c:\user\path\Miniconda3\envs\env\Lib\site-packages\xarray\core\dataarray.py:4237, in DataArray.to_netcdf(self, path, mode, format, group, engine, encoding, unlimited_dims, compute, invalid_netcdf, auto_complex)
   4233 else:
   4234     # No problems with the name - so we're fine!
   4235     dataset = self.to_dataset()
-> 4237 return to_netcdf(  # type: ignore[return-value]  # mypy cannot resolve the overloads:(
   4238     dataset,
   4239     path,
   4240     mode=mode,
   4241     format=format,
   4242     group=group,
   4243     engine=engine,
   4244     encoding=encoding,
   4245     unlimited_dims=unlimited_dims,
   4246     compute=compute,
   4247     multifile=False,
   4248     invalid_netcdf=invalid_netcdf,
   4249     auto_complex=auto_complex,
   4250 )

File c:\user\path\Miniconda3\envs\env\Lib\site-packages\xarray\backends\api.py:2078, in to_netcdf(dataset, path_or_file, mode, format, group, engine, encoding, unlimited_dims, compute, multifile, invalid_netcdf, auto_complex)
   2075 if auto_complex is not None:
   2076     kwargs["auto_complex"] = auto_complex
-> 2078 store = store_open(target, mode, format, group, **kwargs)
   2080 writer = ArrayWriter()
   2082 # TODO: figure out how to refactor this logic (here and in save_mfdataset)
   2083 # to avoid this mess of conditionals

File c:\user\path\Miniconda3\envs\env\site-packages\xarray\backends\netCDF4_.py:457, in NetCDF4DataStore.open(cls, filename, mode, format, group, clobber, diskless, persist, auto_complex, lock, lock_maker, autoclose)
    453     kwargs["auto_complex"] = auto_complex
    454 manager = CachingFileManager(
    455     netCDF4.Dataset, filename, mode=mode, kwargs=kwargs
    456 )
--> 457 return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)

File c:\user\path\Miniconda3\envs\env\Lib\site-packages\xarray\backends\netCDF4_.py:398, in NetCDF4DataStore.__init__(self, manager, group, mode, lock, autoclose)
    396 self._group = group
    397 self._mode = mode
--> 398 self.format = self.ds.data_model
    399 self._filename = self.ds.filepath()
    400 self.is_remote = is_remote_uri(self._filename)

File c:\user\path\Miniconda3\envs\env\Lib\site-packages\xarray\backends\netCDF4_.py:466, in NetCDF4DataStore.ds(self)
    464 @property
    465 def ds(self):
--> 466     return self._acquire()

File c:\user\path\Miniconda3\envs\env\Lib\site-packages\xarray\backends\netCDF4_.py:460, in NetCDF4DataStore._acquire(self, needs_lock)
    459 def _acquire(self, needs_lock=True):
--> 460     with self._manager.acquire_context(needs_lock) as root:
    461         ds = _nc4_require_group(root, self._group, self._mode)
    462     return ds

File c:\user\path\Miniconda3\envs\env\Lib\contextlib.py:141, in _GeneratorContextManager.__enter__(self)
    139 del self.args, self.kwds, self.func
    140 try:
--> 141     return next(self.gen)
    142 except StopIteration:
    143     raise RuntimeError("generator didn't yield") from None

File c:\user\path\Miniconda3\envs\env\Lib\site-packages\xarray\backends\file_manager.py:199, in CachingFileManager.acquire_context(self, needs_lock)
    196 @contextlib.contextmanager
    197 def acquire_context(self, needs_lock=True):
    198     """Context manager for acquiring a file."""
--> 199     file, cached = self._acquire_with_cache_info(needs_lock)
    200     try:
    201         yield file

File c:\user\path\Miniconda3\envs\env\Lib\site-packages\xarray\backends\file_manager.py:217, in CachingFileManager._acquire_with_cache_info(self, needs_lock)
    215     kwargs = kwargs.copy()
    216     kwargs["mode"] = self._mode
--> 217 file = self._opener(*self._args, **kwargs)
    218 if self._mode == "w":
    219     # ensure file doesn't get overridden when opened again
    220     self._mode = "a"

File src\\netCDF4\\_netCDF4.pyx:2521, in netCDF4._netCDF4.Dataset.__init__()

File src\\netCDF4\\_netCDF4.pyx:2158, in netCDF4._netCDF4._ensure_nc_success()

PermissionError: [Errno 13] Permission denied: 'c:\\user\\path\\test.nc'



---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[5], line 1
----> 1 xr.open_dataarray('test.nc')

File c:\user\path\Miniconda3\envs\env\Lib\site-packages\xarray\backends\api.py:956, in open_dataarray(filename_or_obj, engine, chunks, cache, decode_cf, mask_and_scale, decode_times, decode_timedelta, use_cftime, concat_characters, decode_coords, drop_variables, create_default_indexes, inline_array, chunked_array_type, from_array_kwargs, backend_kwargs, **kwargs)
    774 def open_dataarray(
    775     filename_or_obj: T_PathFileOrDataStore,
    776     *,
   (...)    796     **kwargs,
    797 ) -> DataArray:
    798     """Open an DataArray from a file or file-like object containing a single
    799     data variable.
    800 
   (...)    953     open_dataset
    954     """
--> 956     dataset = open_dataset(
    957         filename_or_obj,
    958         decode_cf=decode_cf,
    959         mask_and_scale=mask_and_scale,
    960         decode_times=decode_times,
    961         concat_characters=concat_characters,
    962         decode_coords=decode_coords,
    963         engine=engine,
    964         chunks=chunks,
    965         cache=cache,
    966         drop_variables=drop_variables,
    967         create_default_indexes=create_default_indexes,
    968         inline_array=inline_array,
    969         chunked_array_type=chunked_array_type,
    970         from_array_kwargs=from_array_kwargs,
    971         backend_kwargs=backend_kwargs,
    972         use_cftime=use_cftime,
    973         decode_timedelta=decode_timedelta,
    974         **kwargs,
    975     )
    977     if len(dataset.data_vars) != 1:
    978         if len(dataset.data_vars) == 0:

File c:\user\path\Miniconda3\envs\env\Lib\site-packages\xarray\backends\api.py:731, in open_dataset(filename_or_obj, engine, chunks, cache, decode_cf, mask_and_scale, decode_times, decode_timedelta, use_cftime, concat_characters, decode_coords, drop_variables, create_default_indexes, inline_array, chunked_array_type, from_array_kwargs, backend_kwargs, **kwargs)
    728     kwargs.update(backend_kwargs)
    730 if engine is None:
--> 731     engine = plugins.guess_engine(filename_or_obj)
    733 if from_array_kwargs is None:
    734     from_array_kwargs = {}

File c:\user\path\Miniconda3\envs\env\Lib\site-packages\xarray\backends\plugins.py:199, in guess_engine(store_spec)
    191 else:
    192     error_msg = (
    193         "found the following matches with the input file in xarray's IO "
    194         f"backends: {compatible_engines}. But their dependencies may not be installed, see:\n"
    195         "https://docs.xarray.dev/en/stable/user-guide/io.html \n"
    196         "https://docs.xarray.dev/en/stable/getting-started-guide/installing.html"
    197     )
--> 199 raise ValueError(error_msg)

ValueError: did not find a match in any of xarray's currently installed IO backends ['netcdf4', 'h5netcdf', 'scipy']. Consider explicitly selecting one of the installed engines via the ``engine`` parameter, or installing additional IO dependencies, see:
https://docs.xarray.dev/en/stable/getting-started-guide/installing.html
https://docs.xarray.dev/en/stable/user-guide/io.html

Anything else we need to know?

Critical Issues:

• Data Loss: This is a data corruption bug. Users lose their original files when to_netcdf() "fails."

• Persistent File Locks: File handles remain locked even after closing Jupyter notebooks, which breaks normal multi-notebook workflows.

• Misleading Error Messages: The "IO backends not found" error makes users think their conda environment is broken (I use all three engines suggested by the error message all the time, independently from xarray), when the real issue is that the file xarray tries to read has been truncated to 0 bytes.

• Impact: This makes xarray unsuitable for collaborative analysis or multi-notebook workflows, which are fundamental use cases in data science. Note: the bug affects both dataarrays and datasets.

Suggested Improvements:

• Implement atomic writes (write to temp file, then rename)
• Better file handle management in Jupyter environments
• Detect 0-byte files and provide clear error messages
• Consider adding a force=True parameter for overwriting locked files

Environment

Start with xarray/backends/api.py, backends/netCDF4_.py, and backends/file_manager.py, following the to_netcdf() path shown in the traceback. Reproduce the locked-file overwrite in the Windows multi-notebook scenario; done means a PermissionError leaves the existing NetCDF file unchanged rather than truncating it.