intake / intake/intake-stac

Slow creation with dynamic STAC catalogs

Open
#140 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
113
Forks
25
PR merge metrics
No merged PRs in 30d

Description

In https://github.com/intake/intake-stac/blob/b1451497a6ea40b939f340e397fafad082fcc47a/intake_stac/catalog.py#L123, intake-stac will recurse into child objects. This will end up making many HTTP requests for large, dynamic STAC catalogs served over a STAC API like https://planetarycomputer.microsoft.com/api/stac/v1.

I believe that all the necessary information is provided at the https://planetarycomputer.microsoft.com/api/stac/v1/collections endpoint. https://github.com/stac-utils/pystac-client handles all the logic for interacting with STAC APIs efficiently (it has subclasses for `pystac.Collection`, etc.). Something like

```python
pystac_client.Client.open("https://planetarycomputer.microsoft.com/api/stac/v1/").get_collections()
```

should efficiently get the child *collections* for APIs that implement the `/collections` endpoint (I'm unsure about child *catalogs*; we don't use them).

Might have some overlap with https://github.com/intake/intake-stac/issues/66.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at intake_stac/catalog.py around line 123 and inspect how child objects are recursively loaded. Compare that behavior with pystac_client.Client.open(...).get_collections() against the Planetary Computer STAC API, and review the related issue #66. Done means dynamic catalogs can obtain child collections efficiently without making unnecessary HTTP requests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.