intake / intake/intake-esm

Allow collection to be on file system while catalog is in the cloud

Open
#351 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
164
Forks
54
PR merge metrics
No merged PRs in 30d

Description

When using an esm_collection from file system:

`intake.open_esm_datastore('/home/dkrz/k204210/intake-esm/esm-collections/dkrz_era5_disk_grb_fromcloud.json')`

with url as `catalog_file` entry value:

```
cat /home/dkrz/k204210/intake-esm/esm-collections/dkrz_era5_disk_grb_fromcloud.json
...
"catalog_file": "https://swift.dkrz.de/v1/dkrz_a44962e3ba914c309a7421573a6949a6/intake-esm/dkrz_era5_disk_grb.csv.gz",
```
results in
```
Unable to find: /home/dkrz/k204210/intake-esm/esm-collections/https:/swift.dkrz.de/v1/dkrz_a44962e3ba914c309a7421573a6949a6/intake-esm/dkrz_era5_disk_grb.csv.gz
```

I think there is no reason why this should not work.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the issue with intake.open_esm_datastore and the local JSON file whose catalog_file points to the Swift URL. Trace how catalog_file is resolved when opening an esm_collection; done means a local collection can load its remote catalog without prepending the local collection directory.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.