fsspec / fsspec/kerchunk

Feature Request for MultiZarrToZarr to test whether input chunk sizes are identical

Open
#136 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
366
Forks
96
Avg merge
24m
Merged PRs (30d)
1

Description

I have input zarr datasets which feature identical chunk sizes except for the unlimited dimension, which is time. Each of my input datasets has trailing chunks which differ from one another. I found that MultiZarrToZarr was able to create a new output json from these inputs. The time chunks of varying size then made the resulting combined dataset unable to be loaded in full by xarray, which raised a long, hard-to-parse error traceback.

I was surprised to find that despite being unable to be load the full dataset or a full time series into Xarray, the resulting zarr dataset attributes could be queried by Xarray without issue. Individual time steps could also be loaded, as could time series which were only drawn from one of the input datasets.

I understand that there are efforts being investigated to support unequal chunk sizes, but for now it might be helpful to add in a test to MultiZarrToZarr to inform the user that input datasets need to have identical chunks (well, apart from the last one).

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the MultiZarrToZarr implementation and its existing tests. Reproduce the case with input datasets whose trailing time chunks differ, then add coverage for the validation; done means incompatible chunk sizes are detected with a clear user-facing error while identical chunks, apart from the final one, continue to work.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.