fsspec / fsspec/kerchunk

Tree reduction problem

Open
#299 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
366
Forks
96
Avg merge
24m
Merged PRs (30d)
1

Description

Following this example I'm trying to replicate the tree reduction approch with the Copernicus Global Land products TOC 300m.
In this specific case I've reduced the test to a simple combination of the archive along the time dimension (that's contained in the the filename). To avoid other issues I've previously added the time dimension.

Using the tree reduction approach I get a wrongly structured Dataset that has a single dimension on the concat_dim ( in my case a time coordinate ).
With the simpler approch I've no problem; this bring me to think that the issue isn't coming from the data.

During the computation I get this warning:

UserWarning: Concatenated coordinate 'time' contains less than expectednumber of values across the datasets: [0]
  warnings.warn(

On top of this issue if I try to retreat the time value from the file name throught the coo_map this fails once he try to aggregate the data.

Here the code I'm using even if, as is super simple, I would say that nothing is wrong with in it :
A small sub sample of the original data can be feached here

local_fs = LocalFileSystem()
input_nc =   r"tmp/S3/time/cgl_TOC_*_X2[1-2]Y0[5-6]_*A_*.nc"
input_json = r"tmp/S3/time/cgl_TOC_*_X2[1-2]Y0[5-6]_*A_*.json"
final_output = r"tmp/S3/ensamble.json"

def process(ith_path, outputfile, storage_options_in={}, storage_options_out={}):
    transformer = kerchunk.hdf.SingleHdf5ToZarr(ith_path, inline_threshold=300)
    refs = transformer.translate()
    with fsspec.open(outputfile, mode="wt", **storage_options_out) as f:
        ujson.dump(refs, f)

scan = glob.iglob(
    input_nc, recursive=True
)

infilenames = []
outfilenames = []

for ith_path in scan:
    if not os.path.isfile(ith_path.replace(".nc", ".json")):
        out_path = os.path.join(
            os.path.dirname(ith_path),
            os.path.basename(ith_path).replace(".nc", ".json"),
        )
        outfilenames.append(out_path)
        infilenames.append(ith_path)

tasks = [dask.delayed(process)(u, o) for u, o in zip(infilenames, outfilenames)]
dask.compute(tasks)

json_list = sorted(
    glob.iglob(input_json, recursive=True)
)
json_split = np.array_split(
    np.array(json_list), 2
)
json_split = [list(x) for x in json_split]

def multi_multizarr(json_list):
    mzz = MultiZarrToZarr(
        json_list,
        remote_protocol="file",
        concat_dims=["time"],
        identical_dims=["lat", "lon"],
    )
    return mzz.translate()

jsons = []
for json in json_split:
    x = dask.delayed(multi_multizarr)(json)
    jsons.append(x)

d = dask.delayed(multi_multizarr)(jsons)
d.compute()

d = multi_multizarr(json_list)

with local_fs.open(final_output, "wb") as f:
    f.write(ujson.dumps(d, indent=4).encode())

# Data 
ds = xr.open_dataset(
        "reference://",
        engine="zarr",
        backend_kwargs={
            "storage_options": {
                "fo": final_output,
            },
            "consolidated": False,
        },
    )
ds

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the tree reduction with the issue's sample files and the shown MultiZarrToZarr code, then compare it with the simpler aggregation that works. Inspect the warning about the concatenated time coordinate and the coo_map failure during aggregation. Done means the combined dataset has the expected time dimension and coordinates for the archive.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.