Different behaviour between engines in xarray.Dataset.to_netcdf
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
What happened:
Depending on the engine used writing a xarray.Dataset the behaviour can change notably using engine='scipy' will overwrite the file even if another file is using it. I am not sure this is a pure `xarray problem but would nice to be consistent ....
What you expected to happen:
All engines throw an error if the file is being used
Minimal Complete Verifiable Example:
Open 1 terminal and run:
import fasteners
lock = fasteners.InterProcessLock('test.nc')
lock.acquire()
This is just to create a locked file, you can do this in another manner.
This is the bug:
import xarray as xr
da = xr.DataArray(
[1, 2, 3, 4],
dims=['lat'],
name='lat'
)
ds = da.to_dataset()
ds.to_netcdf('test.nc', engine='netcdf4')
-> PermissionError: [Errno 13] Permission denied: b'/XXXX/XXX/bug.nc'
ds.to_netcdf('.nc', engine='h5netcdf')
-> OSError: Unable to create file (unable to lock file, errno = 35, error message = 'Resource temporarily unavailable')
ds.to_netcdf('.nc', engine='scipy')
-> no output (Saves file with no problems??)
Anything else we need to know?:
Environment:
Output of xr.show_versions()
INSTALLED VERSIONS
commit: None
python: 3.8.5 (default, Sep 4 2020, 02:22:02)
[Clang 10.0.0 ]
python-bits: 64
OS: Darwin
OS-release: 20.2.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.UTF-8
libhdf5: 1.12.0
libnetcdf: 4.7.4
xarray: 0.16.2
pandas: 1.2.0
numpy: 1.19.4
scipy: 1.5.4
netCDF4: 1.5.5.1
pydap: None
h5netcdf: 0.8.1
h5py: 3.1.0
Nio: None
zarr: None
cftime: 1.3.0
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2020.12.0
distributed: 2020.12.0
matplotlib: 3.3.3
cartopy: None
seaborn: None
numbagg: None
pint: None
setuptools: 51.0.0.post20201207
pip: 20.3.3
conda: None
pytest: 6.2.1
IPython: 7.19.0
sphinx: None
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at xarray.Dataset.to_netcdf and reproduce the reported behavior with the netcdf4, h5netcdf, and scipy engines using the provided locked-file example. Trace the engine-specific write paths and identify where regression coverage belongs. Done means the intended behavior for a locked output file is consistent across the affected engines and is covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100