pydata / pydata/xarray

`encoding['chunksizes']` can become stale after `isel()` removes a dimension

Open
#11,028 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

When using isel() to select a single index along a dimension, the variable's encoding['chunksizes'] is preserved unchanged.
This creates a mismatch where the data has N-1 dimensions but the encoding still contains N-dimensional chunksizes.

Minimal example
import tempfile
from pathlib import Path

import numpy as np
import xarray as xr

with tempfile.TemporaryDirectory() as tmpdir:
  original_path = Path(tmpdir) / 'original.nc'
  resaved_path = Path(tmpdir) / 'resaved.nc'

  # Create a chunked netCDF file
  data = np.random.randint(0, 255, size=(5, 10, 20), dtype=np.uint8)
  ds = xr.Dataset({'images': (['time', 'y', 'x'], data)})
  ds['images'].encoding = {'chunksizes': (1, 10, 20)}
  ds.to_netcdf(original_path, engine='h5netcdf')

  # Load it back
  loaded = xr.open_dataset(original_path, engine='h5netcdf')
  print(f"Loaded shape: {loaded.images.shape}")
  print(f"Loaded chunksizes: {loaded.images.encoding.get('chunksizes')}")

  # Use isel to select single index, removing 'time' dimension
  selected = loaded.isel(time=2)
  print(f"\nAfter isel shape: {selected.images.shape}")
  print(f"After isel chunksizes: {selected.images.encoding.get('chunksizes')}")

  # MISMATCH: data is 2D but chunksizes is still 3D
  print(f"\n*** Data has {selected.images.ndim} dims, chunksizes has {len(selected.images.encoding['chunksizes'])} dims ***")

  # Save and reload to see what xarray does with the mismatched encoding
  selected.to_netcdf(resaved_path, engine='h5netcdf')
  reloaded = xr.open_dataset(resaved_path, engine='h5netcdf')
  print(f"\nAfter save/reload:")
  print(f"Shape: {reloaded.images.shape}")
  print(f"Chunksizes: {reloaded.images.encoding.get('chunksizes')}")
Output:
Loaded shape: (5, 10, 20)
Loaded chunksizes: (1, 10, 20)

After isel shape: (10, 20)
After isel chunksizes: (1, 10, 20)

*** Data has 2 dims, chunksizes has 3 dims ***

After save/reload:
Shape: (10, 20)
Chunksizes: None

Our use case:

We work with large chunked netCDF files and have workflows that select subsets before re-saving.
For example, selecting the best image from a stack of images removes a dimension.
Our save code reads encoding['chunksizes'] to preserve the user's chunking preferences, but after isel() removes a dimension, the encoding no longer matches the data.

What we find complicated:

The encoding dict seems to serve two purposes:

  1. Source metadata - "this is how the data was stored in the file you loaded"
  2. Output preferences - "this is how the data should be written when you save"

After isel(), the encoding reflects the source structure, but we need output structure for saving.
We can't easily distinguish "user explicitly set this" from "this was copied from the source file and is now stale."

What we've done:

We detect when len(chunksizes) != ndim and recompute chunksizes with a warning.
But it feels like something that could be handled more systematically.

Questions:

  1. Is this the expected behavior?
  2. Should isel() (or other dimension-reducing operations) update dimension-dependent encoding fields?
  3. Is there a recommended pattern for handling this?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the minimal example through isel() and to_netcdf(), then trace how dimension-dependent encoding is copied during dimension-reducing operations. Compare the resulting encoding with the selected data dimensions and determine the expected behavior for stale chunksizes; done means the behavior is covered by an appropriate regression test and documented or resolved consistently.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.