Extend the ESM collection specification to support catalogs containing datasets in GRIB format
- Dominant language
- Python
- Stars
- 164
- Forks
- 54
- PR merge metrics
- No merged PRs in 30d
Description
**Is your feature request related to a problem? Please describe.**
Currently, the only valid data formats supported by intake-esm are `netcdf` and `zarr`. As a result, it's not possible to use intake-esm when working with datasets in grib format.
**Describe the solution you'd like**
We should consider extending the esm collection specification: https://github.com/intake/intake-esm/blob/main/docs/source/explanation/esm-collection-spec.md#assets-object to allow catalogs to contain datasets in grib format.
**Additional context**
I'm copying and pasting the comment in https://github.com/intake/intake-esm/issues/66#issuecomment-1061610199
I made one for dkrz era5 data which is on our HPC's disk storage. That means, the data access also only works on Mistral.
```
dkrz_cdp=intake.open_catalog("https://swift.dkrz.de/v1/dkrz_a44962e3ba914c309a7421573a6949a6/intake-esm/dkrz_data-pool_cloudcatalog.yaml")
esm_dkrz_era=dkrz_cdp.dkrz_era5_disk_grb_fromcloud
```
should work.
However, it is based on raw `grb` data and `intake-esm` does not really support `grb`. We would need a default engine [here](https://github.com/intake/intake-esm/blob/40fe3a7808c14af77d96cfc932b9cebaa6beba10/intake_esm/source.py#L18) for grib. The one i use is `cfgrib`. You may say that users can provide that but since there is no `grib_kwargs`, I cannot for example merge `grb` assets with `netcdf` assets. Which is why I say there is no `grb` support in intake-esm.
Best,
Fabi
_Originally posted by @wachsylon in https://github.com/intake/intake-esm/issues/66#issuecomment-1061610199_
Contributor guide
Research direction
Read the assets object section of docs/source/explanation/esm-collection-spec.md and inspect intake_esm/source.py, especially the engine handling referenced in the issue. Determine how GRIB catalogs and grib_kwargs should be represented, including mixed GRIB and NetCDF assets. Done means the specification and loading behavior support GRIB datasets consistently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100