pydata / pydata/xarray

Writing integer data in netcdf3 (or classic) is not always lazy

Open
#10,826 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug topic-backends
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

Two steps of the netcdf3 encoding process are triggering unexpected computation in some specific cases. This happens when the function xarray.backend.netcdf4.encode_nc3_variable is called. This concerns saving to netcdf with scipy, but also with netcdf4 when format='NETCDF4_CLASSIC' is required. In both sub-issues, passing compute=False has no impact.

Issue A

Let's say one has a dataset with a dask-backed data var which is a count of days ( ex: yearly series of the count of days over 30°C in a region). Units are denoted as "days" and the dtype of the data is "int32". When we save to netcdf with format='NETCDF4_CLASSIC', the computation is triggered here:
https://github.com/pydata/xarray/blob/e0ad9526166f9e6b94f6059ab5cf8513ee7aad2d/xarray/backends/netcdf3.py#L106-L118

My guess is that this method was written with coordinates in mind, i.e. variable which are almost never using dask or at least are pretty light. In my situation here, the data variable can be quite heavy, requiring a lot of computation I don't want to trigger twice.

I'm not sure the process is needed for timedelta variables however. Maybe the problem could be reduced by changing the test with coding.variables._is_time_like(units) == 'timedelta'. The issue would still happen for datetime data variables.

May be the np.where line could be applied without checking for mask.any() ? If mask.any() is False, then the np.where is a no-op anyway, but at least the computation is not triggered and the result is a dask array.

Issue B

The encoding process then calls coerce_nc3_dtype. When non-netcdf3 integer dtypes are encountered, the data is casted and the verification process triggers computation.
https://github.com/pydata/xarray/blob/e0ad9526166f9e6b94f6059ab5cf8513ee7aad2d/xarray/backends/netcdf3.py#L77-L85

Solutions

For issue A, I had some ideas, but I don't have any for issue B. Maybe in the case of dask-backed arrays, a warning could be raised instead of the checking ? Like "Data was in non-supported integer type {dtype_in} and was converted to {dtype_out}. This could lead to errors.".

For users, the simplest workaround for issue A is to convert to float and for issue B is to convert to one of int32, int16 or int8.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at xarray/backend/netcdf3.py, especially encode_nc3_variable and coerce_nc3_dtype, and trace both Issue A and Issue B with dask-backed integer data and compute=False. Compare the scipy and NETCDF4_CLASSIC paths. Done means integer NetCDF3 encoding no longer triggers unexpected eager computation while preserving the required dtype and encoding behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.