Read json dictionary of references instead of list of json references
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 366
- Forks
- 96
- Avg merge
- 24m
- Merged PRs (30d)
- 1
Description
I am looking to convert the entire NOAA GEFS Retrospective dataset into a Zarr using kerchunk. Currently, the workflow generates a reference file for each GRIB2 file for one year per variable with all ensembles. It consolidates these files into one json. The consolidated json file for each year per variable will be consolidated into one json then converted to a Zarr. The problem is that the NOAA GEFS Retrospective dataset consists of individual files for each day for each variable for each ensemble, so "gen_json" generates millions of reference files. Is it possible to combine the references with "MultiZarrtoZarr" or "auto_dask" from a dictionary instead of a list of json files? This way only one file is written for all the reference files instead of writing individual files. Since, "auto_dask" and "MultiZarrtoZarr" accept a dictionary would the code below work as an input?
out = {}
for f in file_url:
tmp = scan_grib(f, common, so)
outf = f'{json_dir}_{variable}.json'
for i, msg in enumerate(tmp):
outf = f'{json_dir}_{variable}_{msg}.json'
out[i] = msg
fs_write = fsspec.filesystem('')
with fs_write.open('reference_json.json, "w") as f:
json.dump(out, f)
I've tried to iterate over each item in the dictionary within "MultiZarrtoZarr" but this did not work. I also tried turning the resulting dictionary into a list: list(out.item()). This also did not work.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names gen_json, MultiZarrtoZarr, auto_dask, and scan_grib but no repository file or test. Start by tracing how these entry points accept and emit reference collections, then compare the list and dictionary shapes; done means a confirmed supported input or a scoped implementation path with coverage for the NOAA-style workload.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100