geopython / geopython/pygeoapi

HateoasProvider doesn't support STAC best-practices and breaks with shared-folder item layouts

Open
#2,404 0 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Python
Stars
624
Forks
326
Avg merge
8h 25m
Merged PRs (30d)
2

Description

USGS recently restructured their [stac](https://code.usgs.gov/wma/nhgf/stac/-/tree/main?ref_type=heads) repository and in doing so, temporarily broke their production version of [pygeoapi](https://api.water.usgs.gov/gdp/pygeoapi). They used to store each Item in its own uniquely-named subdirectory (`/.json`), but restructured the repository to group Items under a shared `items/` directory instead (`items/.json`).

This change was driven by the [catalog-layout best practices](https://github.com/radiantearth/stac-spec/blob/master/best-practices.md#catalog-layout) document, which states that a one-subdirectory-per-Item layout is only recommended when there are sidecar files alongside the Item. That layout should specifically be avoided when it would "regularly lead to a single Item in a directory," which was the case here, since these Items have no sidecar files.

The [HateoasProvider.get_data_path](https://github.com/geopython/pygeoapi/blob/master/pygeoapi/provider/hateoas.py) builds `rel: "item"` links by splitting each source `catalog.json`/`collection.json` link's `href` on `/` and unpacking exactly three segments:

```python
unused, path_ending, entry_type = link.split('/')
```

`path_ending` is then used as both the `href` and `title` for the item link. This assumes an Item's unique identifier is always the second segment. The HATEOAS provider worked correctly with the old structure, but with the new structure, `path_ending` resolves to the constant folder name `items` for every Item, and the real unique filename (the third segment) is discarded (shown below).

Image

**Steps to Reproduce**

1. Serve a `collection.json` via the HATEOAS/STAC provider where Item hrefs follow `/.json`. Item links resolve correctly.
2. Change the layout so Item hrefs follow `items/.json` instead.
3. Request the collection: `GET /stac/stac-collection//?f=json`.
4. All `rel: "item"` links are now identical.

Or check out USGS's Pygeoapi development instance: https://labs-beta.waterdata.usgs.gov/api/gdp/pygeoapi/stac/stac-collection/nlcd/nlcd-FctImp

**Expected behavior**

Each `rel: "item"` link should have a unique `href`/`title` derived from the Item's actual filename, regardless of directory depth or whether Items share a parent directory.

**Screenshots/Tracebacks**

No exception raised. 40 Items all produce the same link:

**Environment**

- OS: Windows
- Python version: 3.12.13

**Additional context**

Suggested fix: derive the identifier from the basename of the link instead of a fixed-position segment.

```python
for link in link_href_list:
unused, path_ending, entry_type = link.split('/')
newpath = os.path.join(baseurl, urlpath, path_ending).replace('\\', '/') # noqa

if entry_type == 'catalog.json':
child_links.append({
'rel': 'child',
'href': newpath,
'type': 'application/json',
'created': "-",
'entry:type': 'Catalog'
})
elif entry_type == 'collection.json':
child_links.append({
'rel': 'child',
'href': newpath,
'type': 'application/json',
'created': "-",
'entry:type': 'Collection'
})
else:
item_id = os.path.splitext(entry_type)[0]
itempath = os.path.join(newpath, item_id).replace('\\', '/')
child_links.append({
'rel': 'item',
'href': itempath,
'title': item_id,
'created': "-",
'entry:type': 'Item'
})
```

Suggested fix for `get_data_path`'s resolution cascade: add a fallback attempt for a flat sibling file, tried after the existing per-item-subdirectory attempt fails.

```python
except Exception:
try:
filename = os.path.basename(data_path)
jsondata = _get_json_data(f'{data_path}/{filename}.json')
resource_type = 'Assets'
except Exception:
try:
jsondata = _get_json_data(f'{data_path}.json')
resource_type = 'Assets'
except Exception:
msg = f'Resource does not exist: {data_path}'
LOGGER.error(msg)
raise ProviderNotFoundError(msg)
```

Backward-compatible with the existing `/.json` layout, and also correctly handles `items/.json` grouping or any other nesting depth. Happy to provide before/after `collection.json` files and rendered output if useful.

Contributor guide

Open the contributing guide

Research direction

Start in pygeoapi/provider/hateoas.py, especially HateoasProvider.get_data_path and its fixed-position link parsing and path-resolution cascade. Reproduce the collection request with both /.json and items/.json layouts, then verify that each rel: "item" link uses the actual filename and that both layouts resolve successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.