pydata / pydata/xarray

set encoding parameters in addition to the original encoding

Open
#10,085 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

Is your feature request related to a problem?

When writing to disk with to_netcdf, the encoding argument causes existing encoding to be dropped. This is described in the docs.

What is a good approach to add encoding parameters in addition to the original encoding? e.g.

import rioxarray
import xarray as xr
import numpy as np

# make some random dummy netcdf file

data = np.random.rand(4, 4)
lat = np.linspace(10, 20, 4)
lon = np.linspace(10, 20, 4)

ds = xr.Dataset({"dummy": (["lat", "lon"], data)}, coords={"lat": lat, "lon": lon})

ds.rio.set_spatial_dims("lon", "lat", inplace=True)
ds.rio.write_crs("EPSG:4326", inplace=True)

# note the spatial_ref coordinate
print(ds.dummy)
<xarray.DataArray 'dummy' (lat: 4, lon: 4)> Size: 128B
...
Coordinates:
  * lat          (lat) float64 32B 10.0 13.33 16.67 20.0
  * lon          (lon) float64 32B 10.0 13.33 16.67 20.0
    spatial_ref  int64 8B 0
ds.to_netcdf("test.nc", mode="w")

# read it back in - ok

ds2 = xr.open_dataset("test.nc", decode_coords="all")

print(ds2.dummy)
<xarray.DataArray 'dummy' (lat: 4, lon: 4)> Size: 128B
...
Coordinates:
  * lat          (lat) float64 32B 10.0 13.33 16.67 20.0
  * lon          (lon) float64 32B 10.0 13.33 16.67 20.0
    spatial_ref  int64 8B ...
# now compress

ds2.to_netcdf("test_compressed.nc", mode="w", encoding={"dummy": {"compression": "zstd"}})

# read it back in - drops the spatial_ref

ds3 = xr.open_dataset("test_compressed.nc", decode_coords="all")

print(ds3.dummy)
<xarray.DataArray 'dummy' (lat: 4, lon: 4)> Size: 128B
...
Coordinates:
  * lat      (lat) float64 32B 10.0 13.33 16.67 20.0
  * lon      (lon) float64 32B 10.0 13.33 16.67 20.0

this is because rioxarray stores "grid_mapping" in the encoding.

so what is a nice generic way to specify encoding in addition to the original encoding?

encoding = ds2.dummy.encoding.copy()
encoding["compression"] = "zstd"
ds2.to_netcdf("test_compressed_2.nc", mode="w", encoding={"dummy": encoding})
ValueError: unexpected encoding parameters for 'netCDF4' backend: ['szip', 'zstd', 'bzip2', 'blosc']. Valid encodings are: ...

It seems not possible to pass the original encoding back in (even unmodified) due to additional checks

Describe the solution you'd like

in to_netcdf() be able to specify encoding in addition to the original encoding

Describe alternatives you've considered

No response

Additional context

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in xarray/backends/api.py at the additional encoding-parameter checks linked in the issue, then reproduce the rioxarray example with to_netcdf. The change is complete when callers can add compression settings without losing the original grid_mapping encoding or triggering invalid-backend-parameter errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend-api-design, data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.