fsspec / fsspec/kerchunk

Read json dictionary of references instead of list of json references

Open
#315 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
366
Forks
96
Avg merge
24m
Merged PRs (30d)
1

Description

I am looking to convert the entire NOAA GEFS Retrospective dataset into a Zarr using kerchunk. Currently, the workflow generates a reference file for each GRIB2 file for one year per variable with all ensembles. It consolidates these files into one json. The consolidated json file for each year per variable will be consolidated into one json then converted to a Zarr. The problem is that the NOAA GEFS Retrospective dataset consists of individual files for each day for each variable for each ensemble, so "gen_json" generates millions of reference files. Is it possible to combine the references with "MultiZarrtoZarr" or "auto_dask" from a dictionary instead of a list of json files? This way only one file is written for all the reference files instead of writing individual files. Since, "auto_dask" and "MultiZarrtoZarr" accept a dictionary would the code below work as an input?

out = {}
for f in file_url:
    tmp = scan_grib(f, common, so)
    outf = f'{json_dir}_{variable}.json'
    for i, msg in enumerate(tmp):
        outf = f'{json_dir}_{variable}_{msg}.json'
        out[i] = msg
fs_write = fsspec.filesystem('')
    with fs_write.open('reference_json.json, "w") as f:
        json.dump(out, f)

I've tried to iterate over each item in the dictionary within "MultiZarrtoZarr" but this did not work. I also tried turning the resulting dictionary into a list: list(out.item()). This also did not work.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names gen_json, MultiZarrtoZarr, auto_dask, and scan_grib but no repository file or test. Start by tracing how these entry points accept and emit reference collections, then compare the list and dictionary shapes; done means a confirmed supported input or a scoped implementation path with coverage for the NOAA-style workload.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.