fsspec / fsspec/kerchunk

Can kerchunk return references where the final chunk is smaller?

Open
#436 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
366
Forks
96
Avg merge
24m
Merged PRs (30d)
1

Description

Basically I want to check some assumptions I'm making about concatenation of kerchunk references in VirtualiZarr (for this issue).

I would like to know if when I scan e.g. a single netCDF file with kerchunk, for the returned reference dict:

a) are there files where I will get back more than one chunk entry per variable for one file,
b) if so are there files where instead of all those chunks being the same size, one of them could be smaller (like the final chunk can be in the zarr model)?

In the zarr model (b) is allowed, but that doesn't mean that kerchunk ever actually does it.

If (b) never happens, that's nice because then I can basically just take the shape key in each kerchunk reference dict as referring to every chunk from that variable from that file with no exceptions, and I won't have issues of unknowingly concatenating arrays to become variable-length chunks.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by examining kerchunk's generated reference dictionaries for single netCDF files, focusing on whether variables can have multiple chunk entries and whether the final chunk can be smaller. Compare the observed behavior with the zarr model and determine whether the shape key can describe every chunk; the issue is resolved when cases (a) and (b) are answered from kerchunk behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.