Represent (coordinate) variables "symbolically"
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 366
- Forks
- 96
- Avg merge
- 24m
- Merged PRs (30d)
- 1
Description
I'm working with a GRIB2 file, and am interested in minimizing the size of the references file. Currently, the largest values in the references come from the base64-encoded coordinates that were inlined in the references:
'latitude/0': 'base64:AAAAAACAVkBmZmZ...',
This specific variable (and longitude, step, and perhaps time) can be represented "symbolically" (maybe not the right name), with something like a range(90, -90.1, -0.4).
My questions:
- Does something like this make sense to try?
- Does this instead belong in Zarr instead? It seems more generally useful to compress the size of the data, beyond just what Kerchunk inlines (though I'd still want it in Kerchunk, so that inlined references can benefit from it).
Somewhat annoyingly, there are floating point inaccuracies between what I get from np.arange and what's coming out of cfgrib. But hopefully those can be solved.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No file, entry point, or test is named. Start by reviewing how Kerchunk currently handles inlined coordinate references, then compare the proposed symbolic representation with np.arange and cfgrib outputs. Done would require an agreed representation, handling of floating-point differences, and tests for the affected coordinates.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100