Feature Request for MultiZarrToZarr to test whether input chunk sizes are identical
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 366
- Forks
- 96
- Avg merge
- 24m
- Merged PRs (30d)
- 1
Description
I have input zarr datasets which feature identical chunk sizes except for the unlimited dimension, which is time. Each of my input datasets has trailing chunks which differ from one another. I found that MultiZarrToZarr was able to create a new output json from these inputs. The time chunks of varying size then made the resulting combined dataset unable to be loaded in full by xarray, which raised a long, hard-to-parse error traceback.
I was surprised to find that despite being unable to be load the full dataset or a full time series into Xarray, the resulting zarr dataset attributes could be queried by Xarray without issue. Individual time steps could also be loaded, as could time series which were only drawn from one of the input datasets.
I understand that there are efforts being investigated to support unequal chunk sizes, but for now it might be helpful to add in a test to MultiZarrToZarr to inform the user that input datasets need to have identical chunks (well, apart from the last one).
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the MultiZarrToZarr implementation and its existing tests. Reproduce the case with input datasets whose trailing time chunks differ, then add coverage for the validation; done means incompatible chunk sizes are detected with a clear user-facing error while identical chunks, apart from the final one, continue to work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100