Iterating through catalog items is awkward and slow
- Dominant language
- Python
- Stars
- 113
- Forks
- 25
- PR merge metrics
- No merged PRs in 30d
Description
A common need is getting URLs from item assets within a catalog, which involves iterating over hundreds of items. Here is a quick example:
```python
import satsearch
import intake
bbox = [35.48, -3.24, 35.58, -3.14] # (min lon, min lat, max lon, max lat)
dates = '2010-07-01/2020-08-15'
URL='https://earth-search.aws.element84.com/v0'
results = satsearch.Search.search(url=URL,
collections=['sentinel-s2-l2a-cogs'], # note collection='sentinel-s2-l2a-cogs' doesn't work
datetime=dates,
bbox=bbox,
sortby=['+properties.datetime'])
print('%s items' % results.found())
itemCollection = results.items()
#489 items
```
Initializing the catalog is fast!
```python
%%time
catalog = intake.open_stac_item_collection(itemCollection)
#CPU times: user 3.69 ms, sys: 0 ns, total: 3.69 ms
#Wall time: 3.7 ms
```
Iterating through items is slow. I'm a bit confused by the syntax too. I find myself wanting to use an integer index to get the first item in a catalog (`first_item = catalog[0]`) or simplify the code block, but currentlty below to `hrefs = [item.band.metadata.href for item in catalog]` (currently iterating through catalogs returns item IDs as strings.
```python
%%time
band = 'visual'
hrefs = [catalog[item][band].metadata['href'] for item in catalog]
#CPU times: user 4.6 s, sys: 1.23 ms, total: 4.6 s
#Wall time: 4.61 s
```
As for speed, it only takes microseconds to iterate through the underlaying JSON via sat-stac
```python
%%time
band = 'visual'
hrefs = [i.assets[band]['href'] for i in catalog._stac_obj]
#CPU times: user 684 µs, sys: 0 ns, total: 684 µs
#Wall time: 689 µs
```
@martindurant any suggestions here? I'm a bit perplexed about where the code lives to handle `list(catalog)` or for `item in catalog:` ...
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the implementation behind intake.open_stac_item_collection and the catalog behavior used by list(catalog) and for item in catalog. Compare those paths with the faster catalog._stac_obj iteration, then define completion around convenient item access and avoiding the reported per-item slowdown while preserving asset URL retrieval.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100