intake / intake/intake-xarray

Passing multiple kerchunk sideload files to `open_mfdataset`, not possible with intake

Open
#135 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
76
Forks
36
PR merge metrics
No merged PRs in 30d

Description

Standard intake plugins seem to support glob, `*` or `list` urlpath's, to consume multiple files with `open_mfdataset`. This aproach isn't suitable for the `intake_xarray.xzarr.ZarrSource` plugin since it expects the `(urlpath: "reference://")`, and uses `storage_options::fo` to load the sideload file:

```
driver: intake_xarray.xzarr.ZarrSource
args:
urlpath: "reference://"
storage_options:
fo: "sideload.json"
```

Ideally catalog `fo`, should be able to accept `glob` paths ?

More details:

Having many netcdf files with variable dimensions, we hit the "irregular chunk size between files issue" trying to use kerchunk.
So instead of combining netcdf files, to a single sideload json file, we created a sideload .json for each netcdf file, and let xarray take care of the merge. For our datasets this was good enough, and made working with several months of remote data, possible.

Using xarray `open_mfdataset` directly, it was possible to use multiple jsons. e.g:

```
m_list = []
for js in urls:
with fs.open(js) as f:
m_list.append(fsspec.get_mapper("reference://",
fo=ujson.load(f), remote_protocol="file",
remote_options=so))

ds = xr.open_mfdataset(m_list, engine='zarr',
combine="nested",
backend_kwargs={
"consolidated": False,
},
concat_dim="time")
```

It would have been nice to get rid of this code, and use an intake catalog.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.