ESMSource class for collections in Intake catalogs
- Dominant language
- Python
- Stars
- 164
- Forks
- 54
- PR merge metrics
- No merged PRs in 30d
Description
**Is your feature request related to a problem? Please describe.**
Currently, there doesn't seem to be any source class for [Intake-esm](https://github.com/intake/intake-esm) collections, meaning that any Intake catalogs containing them must use `intake_esm.esm_datastore` as the driver (seen in [Pangeo's climate catalog](https://github.com/pangeo-data/pangeo-datastore/blob/master/intake-catalogs/climate.yaml))
```yaml
plugins:
source:
- module: intake_esm
sources:
cmip6_gcs:
args:
esmcol_obj: "https://storage.googleapis.com/cmip6/pangeo-cmip6.json"
description: 'CMIP6 in Google Cloud Storage'
driver: intake_esm.esm_datastore
metadata: {}
```
This means that accessing these entries directly calls the `intake_esm.esm_datastore` constructor and consequently loads the Intake-esm collection's underlying DataFrame into memory:
```python
In [1]: import intake
In [2]: cat = intake.open_catalog("https://raw.githubusercontent.com/pangeo-data/pangeo-datastore/master/intake-catalogs/climate.yaml")
In [3]: cat["cmip6_gcs"]
Out[3]:
```
This can be a computationally expensive task for larger collections, and in some cases completely unnecessary if we only wish to view the metadata of the collection's entry.
**Describe the solution you'd like**
The implementation of an `ESMSource` class, similar to intake-xarray's [`ZarrSource`](https://github.com/intake/intake-xarray/blob/0772a2b548947cb94d32d3b89f3cb6c2fdb7dd61/intake_xarray/xzarr.py#L4), which would store the initial arguments to create an `esm_datastore`, but wouldn't initialize it until a dedicated method was called:
```python
In [4]: cat["cmip6_gcs"]
Out[4]:
In [5]: cat["cmip6_gcs"].load()
Out[5]:
```
This `ESMSource` could then be supplied as a driver in Intake catalogs, making it substantially faster to crawl catalogs containing ESM collections.
**Describe alternatives you've considered**
The current implementation of ESM collections within Intake catalogs works fine for accessing singular collections; when crawling catalogs with ESM collections, I typically use `cat._entries["some_esm_collection"]` to avoid directly loading the collections. This succeeds in getting the metadata of an ESM collection without opening it, but can be a cumbersome use case when crawling catalogs with mixed entry types.
Contributor guide
Assessment
This issue has not been assessed yet.