Slow creation with dynamic STAC catalogs
- Dominant language
- Python
- Stars
- 113
- Forks
- 25
- PR merge metrics
- No merged PRs in 30d
Description
In https://github.com/intake/intake-stac/blob/b1451497a6ea40b939f340e397fafad082fcc47a/intake_stac/catalog.py#L123, intake-stac will recurse into child objects. This will end up making many HTTP requests for large, dynamic STAC catalogs served over a STAC API like https://planetarycomputer.microsoft.com/api/stac/v1.
I believe that all the necessary information is provided at the https://planetarycomputer.microsoft.com/api/stac/v1/collections endpoint. https://github.com/stac-utils/pystac-client handles all the logic for interacting with STAC APIs efficiently (it has subclasses for `pystac.Collection`, etc.). Something like
```python
pystac_client.Client.open("https://planetarycomputer.microsoft.com/api/stac/v1/").get_collections()
```
should efficiently get the child *collections* for APIs that implement the `/collections` endpoint (I'm unsure about child *catalogs*; we don't use them).
Might have some overlap with https://github.com/intake/intake-stac/issues/66.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at intake_stac/catalog.py around line 123 and inspect how child objects are recursively loaded. Compare that behavior with pystac_client.Client.open(...).get_collections() against the Planetary Computer STAC API, and review the related issue #66. Done means dynamic catalogs can obtain child collections efficiently without making unnecessary HTTP requests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100