pydata / pydata/xarray

Different behaviour between engines in xarray.Dataset.to_netcdf

Open
#4,790 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic-backends
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

What happened:
Depending on the engine used writing a xarray.Dataset the behaviour can change notably using engine='scipy' will overwrite the file even if another file is using it. I am not sure this is a pure `xarray problem but would nice to be consistent ....

What you expected to happen:
All engines throw an error if the file is being used

Minimal Complete Verifiable Example:
Open 1 terminal and run:

import fasteners

lock = fasteners.InterProcessLock('test.nc') 
lock.acquire()

This is just to create a locked file, you can do this in another manner.

This is the bug:

import xarray as xr

da = xr.DataArray(
    [1, 2, 3, 4],
    dims=['lat'],
    name='lat'
)
ds = da.to_dataset()

ds.to_netcdf('test.nc', engine='netcdf4')
-> PermissionError: [Errno 13] Permission denied: b'/XXXX/XXX/bug.nc'

ds.to_netcdf('.nc', engine='h5netcdf')
-> OSError: Unable to create file (unable to lock file, errno = 35, error message = 'Resource temporarily unavailable')

ds.to_netcdf('.nc', engine='scipy')
-> no output (Saves file with no problems??)

Anything else we need to know?:

Environment:

Output of xr.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.8.5 (default, Sep 4 2020, 02:22:02)
[Clang 10.0.0 ]
python-bits: 64
OS: Darwin
OS-release: 20.2.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.UTF-8
libhdf5: 1.12.0
libnetcdf: 4.7.4

xarray: 0.16.2
pandas: 1.2.0
numpy: 1.19.4
scipy: 1.5.4
netCDF4: 1.5.5.1
pydap: None
h5netcdf: 0.8.1
h5py: 3.1.0
Nio: None
zarr: None
cftime: 1.3.0
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2020.12.0
distributed: 2020.12.0
matplotlib: 3.3.3
cartopy: None
seaborn: None
numbagg: None
pint: None
setuptools: 51.0.0.post20201207
pip: 20.3.3
conda: None
pytest: 6.2.1
IPython: 7.19.0
sphinx: None

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at xarray.Dataset.to_netcdf and reproduce the reported behavior with the netcdf4, h5netcdf, and scipy engines using the provided locked-file example. Trace the engine-specific write paths and identify where regression coverage belongs. Done means the intended behavior for a locked output file is consistent across the affected engines and is covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.