Document and decide xlink resolution approach
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 200
- Avg merge
- 4h 56m
- Merged PRs (30d)
- 5
Description
**What:** Based on SPIKE findings, implement the agreed approach for handling xlinks in ISO records.
**If Option A (resolve at harvest time):**
- Add an xlink resolution step before MDTranslator is called
- For each `xlink:href` in the raw ISO XML, make an HTTP GET request to the referenced URL
- Replace the xlink element with the fetched content inline
- Handle failures gracefully: log, continue without the field, do not fail the record
- Add timeout and retry logic appropriate for external service calls
- Test with the Census NGDC endpoints
**If Option B (pass through and accept gaps):**
- Document the known gap in a harvester README or decision record
- Confirm that `fill_placeholders` handles all fields that would be empty due to unresolved xlinks
- Add a warning log when an unresolved xlink is detected
**If Option C (require agencies to use inline values):**
- Draft publisher guidance for resources.data.gov explaining the limitation
- Add a harvest warning (not error) when unresolved xlinks are detected in a record
**Done When:**
- PM has chosen an option
- Chosen option is implemented
- Decision is documented in the harvester repo
**Notes:**
- Option A adds latency per-record. For a WAF with hundreds of records this is significant. Consider whether xlink resolution should be cached across records in the same harvest run.
- The NGDC docucomp endpoints used by Census are the most common xlink targets in federal ISO records. Test specifically with these.
- Option C is only viable if agencies can actually update their ISO records -- not always within their control.
Contributor guide
Assessment
This issue has not been assessed yet.