geopython / geopython/pygeoapi
HateoasProvider doesn't support STAC best-practices and breaks with shared-folder item layouts
- Dominant language
- Python
- Stars
- 624
- Forks
- 326
- Avg merge
- 8h 25m
- Merged PRs (30d)
- 2
Description
USGS recently restructured their [stac](https://code.usgs.gov/wma/nhgf/stac/-/tree/main?ref_type=heads) repository and in doing so, temporarily broke their production version of [pygeoapi](https://api.water.usgs.gov/gdp/pygeoapi). They used to store each Item in its own uniquely-named subdirectory (`/.json`), but restructured the repository to group Items under a shared `items/` directory instead (`items/.json`).
This change was driven by the [catalog-layout best practices](https://github.com/radiantearth/stac-spec/blob/master/best-practices.md#catalog-layout) document, which states that a one-subdirectory-per-Item layout is only recommended when there are sidecar files alongside the Item. That layout should specifically be avoided when it would "regularly lead to a single Item in a directory," which was the case here, since these Items have no sidecar files.
The [HateoasProvider.get_data_path](https://github.com/geopython/pygeoapi/blob/master/pygeoapi/provider/hateoas.py) builds `rel: "item"` links by splitting each source `catalog.json`/`collection.json` link's `href` on `/` and unpacking exactly three segments:
```python
unused, path_ending, entry_type = link.split('/')
```
`path_ending` is then used as both the `href` and `title` for the item link. This assumes an Item's unique identifier is always the second segment. The HATEOAS provider worked correctly with the old structure, but with the new structure, `path_ending` resolves to the constant folder name `items` for every Item, and the real unique filename (the third segment) is discarded (shown below).
**Steps to Reproduce**
1. Serve a `collection.json` via the HATEOAS/STAC provider where Item hrefs follow `/.json`. Item links resolve correctly.
2. Change the layout so Item hrefs follow `items/.json` instead.
3. Request the collection: `GET /stac/stac-collection//?f=json`.
4. All `rel: "item"` links are now identical.
Or check out USGS's Pygeoapi development instance: https://labs-beta.waterdata.usgs.gov/api/gdp/pygeoapi/stac/stac-collection/nlcd/nlcd-FctImp
**Expected behavior**
Each `rel: "item"` link should have a unique `href`/`title` derived from the Item's actual filename, regardless of directory depth or whether Items share a parent directory.
**Screenshots/Tracebacks**
No exception raised. 40 Items all produce the same link:
**Environment**
- OS: Windows
- Python version: 3.12.13
**Additional context**
Suggested fix: derive the identifier from the basename of the link instead of a fixed-position segment.
```python
for link in link_href_list:
unused, path_ending, entry_type = link.split('/')
newpath = os.path.join(baseurl, urlpath, path_ending).replace('\\', '/') # noqa
if entry_type == 'catalog.json':
child_links.append({
'rel': 'child',
'href': newpath,
'type': 'application/json',
'created': "-",
'entry:type': 'Catalog'
})
elif entry_type == 'collection.json':
child_links.append({
'rel': 'child',
'href': newpath,
'type': 'application/json',
'created': "-",
'entry:type': 'Collection'
})
else:
item_id = os.path.splitext(entry_type)[0]
itempath = os.path.join(newpath, item_id).replace('\\', '/')
child_links.append({
'rel': 'item',
'href': itempath,
'title': item_id,
'created': "-",
'entry:type': 'Item'
})
```
Suggested fix for `get_data_path`'s resolution cascade: add a fallback attempt for a flat sibling file, tried after the existing per-item-subdirectory attempt fails.
```python
except Exception:
try:
filename = os.path.basename(data_path)
jsondata = _get_json_data(f'{data_path}/{filename}.json')
resource_type = 'Assets'
except Exception:
try:
jsondata = _get_json_data(f'{data_path}.json')
resource_type = 'Assets'
except Exception:
msg = f'Resource does not exist: {data_path}'
LOGGER.error(msg)
raise ProviderNotFoundError(msg)
```
Backward-compatible with the existing `/.json` layout, and also correctly handles `items/.json` grouping or any other nesting depth. Happy to provide before/after `collection.json` files and rendered output if useful.
Contributor guide
Research direction
Start in pygeoapi/provider/hateoas.py, especially HateoasProvider.get_data_path and its fixed-position link parsing and path-resolution cascade. Reproduce the collection request with both /.json and items/.json layouts, then verify that each rel: "item" link uses the actual filename and that both layouts resolve successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100