Can kerchunk return references where the final chunk is smaller?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 366
- Forks
- 96
- Avg merge
- 24m
- Merged PRs (30d)
- 1
Description
Basically I want to check some assumptions I'm making about concatenation of kerchunk references in VirtualiZarr (for this issue).
I would like to know if when I scan e.g. a single netCDF file with kerchunk, for the returned reference dict:
a) are there files where I will get back more than one chunk entry per variable for one file,
b) if so are there files where instead of all those chunks being the same size, one of them could be smaller (like the final chunk can be in the zarr model)?
In the zarr model (b) is allowed, but that doesn't mean that kerchunk ever actually does it.
If (b) never happens, that's nice because then I can basically just take the shape key in each kerchunk reference dict as referring to every chunk from that variable from that file with no exceptions, and I won't have issues of unknowingly concatenating arrays to become variable-length chunks.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by examining kerchunk's generated reference dictionaries for single netCDF files, focusing on whether variables can have multiple chunk entries and whether the final chunk can be smaller. Compare the observed behavior with the zarr model and determine whether the shape key can describe every chunk; the issue is resolved when cases (a) and (b) are answered from kerchunk behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100