Allow collection to be on file system while catalog is in the cloud
- Dominant language
- Python
- Stars
- 164
- Forks
- 54
- PR merge metrics
- No merged PRs in 30d
Description
When using an esm_collection from file system:
`intake.open_esm_datastore('/home/dkrz/k204210/intake-esm/esm-collections/dkrz_era5_disk_grb_fromcloud.json')`
with url as `catalog_file` entry value:
```
cat /home/dkrz/k204210/intake-esm/esm-collections/dkrz_era5_disk_grb_fromcloud.json
...
"catalog_file": "https://swift.dkrz.de/v1/dkrz_a44962e3ba914c309a7421573a6949a6/intake-esm/dkrz_era5_disk_grb.csv.gz",
```
results in
```
Unable to find: /home/dkrz/k204210/intake-esm/esm-collections/https:/swift.dkrz.de/v1/dkrz_a44962e3ba914c309a7421573a6949a6/intake-esm/dkrz_era5_disk_grb.csv.gz
```
I think there is no reason why this should not work.
Contributor guide
Research direction
Start by reproducing the issue with intake.open_esm_datastore and the local JSON file whose catalog_file points to the Swift URL. Trace how catalog_file is resolved when opening an esm_collection; done means a local collection can load its remote catalog without prepending the local collection directory.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100