[EPIC] Multi-Format Harvest Pipeline - ISO + DCAT-US v1.1 + DCAT-US v3.0
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 200
- Avg merge
- 4h 56m
- Merged PRs (30d)
- 5
Description
**What:** The datagov-harvester must simultaneously ingest three metadata formats: DCAT-US v1.1 (existing data.json files), DCAT-US v3.0 (new standard under M-25-05), and ISO 19115 encoded as ISO 19139 XML (harvested from WAF and document sources). Each format has different field structures, encoding requirements, and downstream expectations. As v3.0 migration accelerates and GeoPlatform coordination grows, the pipeline has several gaps that need investigation and resolution.
**Why:**
- M-25-05 requires DCAT-US v3.0 compliance by September 30, 2026
- ISO harvest sources (WAF, document) are active and must continue to work
- ISO records currently produce DCAT-US v1.1 output via MDTranslator, not v3.0
- v3.0 introduces structured types for `spatial` and `temporal` that differ from v1.1
- GeoPlatform depends on correct spatial, temporal, and ISO-linked records from Data.gov
- Several pipeline behaviors (xlink handling, v3.0 routing, antimeridian edge cases) are undocumented or unconfirmed
**Done When:**
- All spike tickets have documented findings that close their open questions
- ISO records produce v3.0-compatible output in the pipeline
- v3.0 `spatial` (Location object) and `temporal` (PeriodOfTime object) are correctly handled for all source formats
- v3.0 schema_type routing is confirmed and documented
- Known dropped ISO fields are documented in a decision record
- CSW source type is scoped (even if not yet built)
- Identity resolution gap is documented with a recommended approach
**Notes:**
- Ticket order below reflects recommended sequencing: spikes first, then implementation
- Spike findings feed into implementation ticket scoping
- Decisions marked **NEEDS PM INPUT** require product decision before implementation begins
Contributor guide
Assessment
This issue has not been assessed yet.