GSA / GSA/data.gov

Apply dcat_convert.py to MDTranslator output for ISO records

Open
#6,042 0 comments 0 reactions 1 assignee Claimed by @jaredb96 View on GitHub
DCAT-US3 Harvester MD Translator
Dominant language
Python
Stars
1.1k
Forks
200
Avg merge
4h 56m
Merged PRs (30d)
5

Description

**What:** MDTranslator currently produces DCAT-US v1.1 JSON from ISO 19139 XML. As the pipeline moves to v3.0, ISO-sourced records need to produce v3.0-compatible output. The planned `dcat_convert.py` migration script (targeted July 2026) converts v1.1 records to v3.0.

This ticket applies `dcat_convert.py` as a post-MDTranslator step for ISO records, so that pipeline output for ISO sources is v3.0 rather than v1.1.

**Likely scope:**
- Confirm that `dcat_convert.py` is implemented as a callable module (not only a CLI tool)
- Add a post-transform step in `Record.harvest()` that calls `dcat_convert.py` on the MDTranslator output when `schema_type` is `iso19115_1` or `iso19115_2`
- Ensure the upgraded output passes v3.0 validation before sync
- Update `fill_placeholders` and `improve_distributions` steps if any field names or structures changed in v3.0

**Done When:**
- ISO WAF records produce v3.0-compatible JSON after the full harvest pipeline
- Records pass v3.0 schema validation
- The Census concity WAF can be harvested end-to-end with v3.0 output
- v1.1 data.json sources are unaffected

**Notes:**
- Coordinate with whoever is building `dcat_convert.py` to confirm the module interface
- This ticket assumes Option A from the problem statement (MDTranslator + upgrade) and should not begin until PM confirms that approach
- If Option B (native ISO-to-v3.0 transform) is chosen, this ticket is replaced by a larger implementation effort

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.