GSA / GSA/data.gov

Document and decide xlink resolution approach

Open
#6,043 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.1k
Forks
200
Avg merge
4h 56m
Merged PRs (30d)
5

Description

**What:** Based on SPIKE findings, implement the agreed approach for handling xlinks in ISO records.

**If Option A (resolve at harvest time):**
- Add an xlink resolution step before MDTranslator is called
- For each `xlink:href` in the raw ISO XML, make an HTTP GET request to the referenced URL
- Replace the xlink element with the fetched content inline
- Handle failures gracefully: log, continue without the field, do not fail the record
- Add timeout and retry logic appropriate for external service calls
- Test with the Census NGDC endpoints

**If Option B (pass through and accept gaps):**
- Document the known gap in a harvester README or decision record
- Confirm that `fill_placeholders` handles all fields that would be empty due to unresolved xlinks
- Add a warning log when an unresolved xlink is detected

**If Option C (require agencies to use inline values):**
- Draft publisher guidance for resources.data.gov explaining the limitation
- Add a harvest warning (not error) when unresolved xlinks are detected in a record

**Done When:**
- PM has chosen an option
- Chosen option is implemented
- Decision is documented in the harvester repo

**Notes:**
- Option A adds latency per-record. For a WAF with hundreds of records this is significant. Consider whether xlink resolution should be cached across records in the same harvest run.
- The NGDC docucomp endpoints used by Census are the most common xlink targets in federal ISO records. Test specifically with these.
- Option C is only viable if agencies can actually update their ISO records -- not always within their control.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.