Refactor MultiZarrToZarr into multiple functions
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 366
- Forks
- 96
- Avg merge
- 24m
- Merged PRs (30d)
- 1
Description
Problem
MultiZarrToZarr is extremely powerful but rather hard to use.
This is important - kerchunk has been transformative, so we increasingly recommend it as the best way to ingest large amounts of data into the pangeo ecosystem's tools. However that means we should make sure the kerchunk user experience is smooth, so that new users don't get stuck early on.
Part of the problem is that this one MultiZarrToZarr function can do many different things. Contrast with xarray - when combining multiple datasets into one, xarray takes some care to distinguish between a few common cases/concepts (we even have a glossary):
- Concatenation along a single existing dimension. Achieved by
xr.concatwheredimis a str - Concatenation along a single new dimension (optionally providing new coordinates to use along that new dimension). Achieved by
xr.concatwheredimis a set of values - Merging of multiple variables which already share dimensions, first aligned according to their coordinates. Achieved by
xr.merge - "Combining" by order given, which means some ordered combination of concatenation along one or more dimensions and/or merging. Achieved by
xr.combine_nested - "Combining" by coordinate order, which again means some ordered combination of concatenation along one or more dimensions and/or merging, but the order is specified by information in the datasets' coordinates. Achieved by
xr.combine_by_coords
In kerchunk it seems that the recommended way to handle operations resembling all 5 of these cases is through MultiZarrToZarr. It also cannot currently easily handle certain types of multi-dimensional concatenation.
Suggestion
Break up MultiZarrToZarr by defining a set of functions similar to xarray's merge/concat/combine/unify_chunks that consume and produce VirtualZarrStore objects (EDIT: see https://github.com/fsspec/kerchunk/issues/375).
Advantages
- We can replace/deprecate the heavily overloaded and unituitive
coo_mapkwarg (it has 10 possible input types!). Perhaps giving simply an ordered list of coordinate values would be sufficient, and just make it easier for the user to extract the values they want from theVirtualZarrStoreobjects they want to concatenate. - If users need to do something really unusual they can more easily break their problem up into concatenating each array separately (e.g. for concatenating on staggered grids)
- Might generalise to later ZEPs more easily (e.g. understanding variable-length chunks, cc @ivirshup, see https://github.com/fsspec/kerchunk/pull/374)
- Can think of as a refactoring to move some pangeo-forge functionality upstream, reducing redundancy. We shouldn't have 3 completely different designs for multidimensional concatenation in adjacent libraries in the stack.
- These new functions would be more useful as basic primitives for parallelization frameworks to call (e.g. doing tree reduction via dask, beam, or cubed), rather than trying to wrap calls to those frameworks within kerchunk (like
kerchunk.combine.auto_daskdoes).
Questions
- How close can these functions be to xarray's version of
merge/concat/combine? And what can we learn from the design decisions in pangeo-forge-recipesFilePattern? (@cisaacstern @rabernat ) - How close are kerchunk's existing
combine.merge_varsandcombine.concatenate_arraysfunctions to providing this functionality? If the answer is "pretty close", then how much of this issue could be solved via documentation?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading MultiZarrToZarr and the existing combine.merge_vars and combine.concatenate_arrays functions, then compare their behavior with xarray's merge, concat, and combine APIs. The issue leaves the function boundaries, compatibility plan, and documentation-only scope unresolved; done would require an agreed decomposition into VirtualZarrStore operations and corresponding tests or documentation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100