Tree reduction problem
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 366
- Forks
- 96
- Avg merge
- 24m
- Merged PRs (30d)
- 1
Description
Following this example I'm trying to replicate the tree reduction approch with the Copernicus Global Land products TOC 300m.
In this specific case I've reduced the test to a simple combination of the archive along the time dimension (that's contained in the the filename). To avoid other issues I've previously added the time dimension.
Using the tree reduction approach I get a wrongly structured Dataset that has a single dimension on the concat_dim ( in my case a time coordinate ).
With the simpler approch I've no problem; this bring me to think that the issue isn't coming from the data.
During the computation I get this warning:
UserWarning: Concatenated coordinate 'time' contains less than expectednumber of values across the datasets: [0]
warnings.warn(
On top of this issue if I try to retreat the time value from the file name throught the coo_map this fails once he try to aggregate the data.
Here the code I'm using even if, as is super simple, I would say that nothing is wrong with in it :
A small sub sample of the original data can be feached here
local_fs = LocalFileSystem()
input_nc = r"tmp/S3/time/cgl_TOC_*_X2[1-2]Y0[5-6]_*A_*.nc"
input_json = r"tmp/S3/time/cgl_TOC_*_X2[1-2]Y0[5-6]_*A_*.json"
final_output = r"tmp/S3/ensamble.json"
def process(ith_path, outputfile, storage_options_in={}, storage_options_out={}):
transformer = kerchunk.hdf.SingleHdf5ToZarr(ith_path, inline_threshold=300)
refs = transformer.translate()
with fsspec.open(outputfile, mode="wt", **storage_options_out) as f:
ujson.dump(refs, f)
scan = glob.iglob(
input_nc, recursive=True
)
infilenames = []
outfilenames = []
for ith_path in scan:
if not os.path.isfile(ith_path.replace(".nc", ".json")):
out_path = os.path.join(
os.path.dirname(ith_path),
os.path.basename(ith_path).replace(".nc", ".json"),
)
outfilenames.append(out_path)
infilenames.append(ith_path)
tasks = [dask.delayed(process)(u, o) for u, o in zip(infilenames, outfilenames)]
dask.compute(tasks)
json_list = sorted(
glob.iglob(input_json, recursive=True)
)
json_split = np.array_split(
np.array(json_list), 2
)
json_split = [list(x) for x in json_split]
def multi_multizarr(json_list):
mzz = MultiZarrToZarr(
json_list,
remote_protocol="file",
concat_dims=["time"],
identical_dims=["lat", "lon"],
)
return mzz.translate()
jsons = []
for json in json_split:
x = dask.delayed(multi_multizarr)(json)
jsons.append(x)
d = dask.delayed(multi_multizarr)(jsons)
d.compute()
d = multi_multizarr(json_list)
with local_fs.open(final_output, "wb") as f:
f.write(ujson.dumps(d, indent=4).encode())
# Data
ds = xr.open_dataset(
"reference://",
engine="zarr",
backend_kwargs={
"storage_options": {
"fo": final_output,
},
"consolidated": False,
},
)
ds
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the tree reduction with the issue's sample files and the shown MultiZarrToZarr code, then compare it with the simpler aggregation that works. Inspect the warning about the concatenated time coordinate and the coo_map failure during aggregation. Done means the combined dataset has the expected time dimension and coordinates for the archive.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100