Control chunksize of the underlying zarrdata
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 366
- Forks
- 96
- Avg merge
- 24m
- Merged PRs (30d)
- 1
Description
I would appreciate to have more control over the way Kerchunk is writing "refs" -especially control the chunking.
Context:
I previously used fsspec and kerchunk to store my data while continously exanding my dataset.
My data always has the same dimensionality of course and even the same coordinetes except one dimension: "release"
When using class SingleHdf5ToZarr in Kerchunk I have no control over the zarr group/store created
I think it is because this part of the init is hardcoded and not mutable through any methods:
self.store = {}
self._zroot = zarr.group(store=self.store, overwrite=True)
My data files have the same coordinate size : datafile_shape=(1,n2,n3,n4)
When running SingleHdf5ToZarr(...).translate() on my old data, I get back data with some arbitrary chunksize (1,n_c2,n_c3,n_c4)
Now that I have updated some dependencies in my env I get another arbitrary chunksize (1,n_c2',n_c3',n_c4')
Here I would actually ideally just have had chunksize = datafile_shape. But the fatal issue is that I can no longer combine new and old data with MultiZarrToZarr. When I try to combine my kerchunk metadata chunks I get:
ValueError: Found chunk size mismatch:
at prefix [my variable name] in iteration 544 (file None)
new chunk: [1, 63, 200, 261]
chunks so far: [1, 42, 133, 174]
Problem in short:
- I have one distributed dataset, that is constantly expanding. Old data is no longer compatible with new data, because my env has changed slightly.
- I experience the arb. chunksize from Kerchunk to deviate between Old and new data, disabling me to combine my dataset properly.
- Ideally I would like to choose the chunksize explicitly like I can in xarray. Then I would prefer my chunksizes to be the same as my datafiles shapes
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by examining SingleHdf5ToZarr.init and the hardcoded zarr.group setup described in the issue, then trace how translate() determines chunk sizes. Define how an explicit chunk shape should be supplied and verify that generated refs preserve it and can be combined by MultiZarrToZarr with matching metadata.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100