fsspec / fsspec/kerchunk

Represent (coordinate) variables "symbolically"

Open
#361 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
366
Forks
96
Avg merge
24m
Merged PRs (30d)
1

Description

I'm working with a GRIB2 file, and am interested in minimizing the size of the references file. Currently, the largest values in the references come from the base64-encoded coordinates that were inlined in the references:

  'latitude/0': 'base64:AAAAAACAVkBmZmZ...',

This specific variable (and longitude, step, and perhaps time) can be represented "symbolically" (maybe not the right name), with something like a range(90, -90.1, -0.4).

My questions:

  1. Does something like this make sense to try?
  2. Does this instead belong in Zarr instead? It seems more generally useful to compress the size of the data, beyond just what Kerchunk inlines (though I'd still want it in Kerchunk, so that inlined references can benefit from it).

Somewhat annoyingly, there are floating point inaccuracies between what I get from np.arange and what's coming out of cfgrib. But hopefully those can be solved.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No file, entry point, or test is named. Start by reviewing how Kerchunk currently handles inlined coordinate references, then compare the proposed symbolic representation with np.arange and cfgrib outputs. Done would require an agreed representation, handling of floating-point differences, and tests for the affected coordinates.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.